u/Dhan295

Trying to understand how teams handle this in practice.

When you deploy a self-hosted open-weight model for an agent (something that makes a bunch of tool calls in a row), who decides it’s ready to go live and what does that check actually look like?

From what I’ve seen, the usual benchmark scores don’t really predict whether a model holds up over a long multi-step run. It can look fine and then fail quietly once it’s live. And the same model behaves differently depending on the runtime and quantization it’s served with, so “passed in testing” and “works in prod” aren’t the same thing.

For people running this for real:

• Is there a real pre-deployment check for self-hosted models, or is it mostly deploy-and-monitor?

• Who owns that gate the ML team, platform/ops, or nobody clearly?

• What do you wish you’d caught before it went live instead of after?

Trying to learn how this works in the real world. What’s your setup?

Honest question. I’m still learning how people do this in real production.

I keep reading that benchmark scores don’t really tell you if a model will hold up as an agent, where it has to make a bunch of tool calls in a row. It can look good on paper and then quietly break halfway through. wrong tool, broken JSON, or it loses track of the task.

I’ve also seen people say the same model behaves differently depending on how it’s served the runtime and the quantization change how reliable it is. So “it worked in testing” doesn’t always mean “it works in prod.”

I haven’t run vLLM in production myself, so I’m trying to understand from people who have:

• How do you decide a model is actually ready to ship as an agent, not just fast?
• Do you have a real check before deploying, or is it more “ship it and watch”?
• What breaks most often once it’s live that you wish you’d caught earlier?

Genuinely just trying to learn how this is done. What does your process look like?