How would you select a LLM model for internal hosting
Something that rarely shows up in MLOps content but is a real part of the job: when an org decides to self-host an LLM, someone needs to figure out which model to actually run and whether the cost makes sense.
It is not "pick the highest benchmark score." It is "what fits on the hardware we can afford and still meets the quality bar."
Example of the tradeoff: Qwen3.5-35B-A3B has 35 billion total parameters but only 3 billion active (MoE). Fits on a single H100. On many tasks it beats Haiku-class API models. Cost: one GPU instance.
GLM 5.2 has 745 billion parameters, about 40 billion active. Open weights, MIT license. Benchmarks within 1 to 3 points of Sonnet 4.6 on coding tasks. But it needs roughly 744 to 890 GB VRAM in FP8. That is 10 to 12 H100s minimum.
So the question becomes: is that marginal quality bump worth 10x the GPU cost? For most use cases, no. For some, yes. That is the analysis.
Tools like AIPerf (github.com/ai-dynamo/aiperf) help with this. You benchmark TTFT, inter-token latency, throughput at different concurrency levels, on your actual hardware, with your actual prompts. Public benchmarks tell you what a model can do in ideal conditions. AIPerf tells you what it does on your infra.
The thinking behind model selection:
- Define what "good enough" means for the use case. Internal doc QA does not need frontier reasoning.
- Benchmark on your workload, not public leaderboards.
- Calculate cost per request, not cost per token. Factor in throughput and concurrency.
- Consider operational overhead. One GPU is simple. A 12-GPU cluster with tensor parallelism is not.
This is the kind of work that makes MLOps different from just knowing tools. Understanding model architectures (dense vs MoE, active vs total parameters), GPU memory math, and cost-quality tradeoffs is part of the job.
Has anyone here gone through a model selection process for internal hosting? Curious what you ended up picking and why.