u/Extension_Key_5970

▲ 5 r/mlops

How would you select a LLM model for internal hosting

Something that rarely shows up in MLOps content but is a real part of the job: when an org decides to self-host an LLM, someone needs to figure out which model to actually run and whether the cost makes sense.

It is not "pick the highest benchmark score." It is "what fits on the hardware we can afford and still meets the quality bar."

Example of the tradeoff: Qwen3.5-35B-A3B has 35 billion total parameters but only 3 billion active (MoE). Fits on a single H100. On many tasks it beats Haiku-class API models. Cost: one GPU instance.

GLM 5.2 has 745 billion parameters, about 40 billion active. Open weights, MIT license. Benchmarks within 1 to 3 points of Sonnet 4.6 on coding tasks. But it needs roughly 744 to 890 GB VRAM in FP8. That is 10 to 12 H100s minimum.

So the question becomes: is that marginal quality bump worth 10x the GPU cost? For most use cases, no. For some, yes. That is the analysis.

Tools like AIPerf (github.com/ai-dynamo/aiperf) help with this. You benchmark TTFT, inter-token latency, throughput at different concurrency levels, on your actual hardware, with your actual prompts. Public benchmarks tell you what a model can do in ideal conditions. AIPerf tells you what it does on your infra.

The thinking behind model selection:

  • Define what "good enough" means for the use case. Internal doc QA does not need frontier reasoning.
  • Benchmark on your workload, not public leaderboards.
  • Calculate cost per request, not cost per token. Factor in throughput and concurrency.
  • Consider operational overhead. One GPU is simple. A 12-GPU cluster with tensor parallelism is not.

This is the kind of work that makes MLOps different from just knowing tools. Understanding model architectures (dense vs MoE, active vs total parameters), GPU memory math, and cost-quality tradeoffs is part of the job.

Has anyone here gone through a model selection process for internal hosting? Curious what you ended up picking and why.

reddit.com
u/Extension_Key_5970 — 2 days ago
▲ 1 r/mlops

How would you catch if Your model is 100% available and 100% wrong at the same time

Something I keep running into: every infra dashboard is green, no errors, latency is fine. But the model is giving bad outputs. Nobody gets paged because technically nothing is broken.

The question that bugged me: if nothing looks broken, how do you know something is wrong?

The thing I learned: infrastructure monitoring and model monitoring answer completely different questions. One tells you the system is up. The other tells you the system is right.

And there is an order to how these failures show up.

Input data shifts first.

Model outputs shift second.

The gap between what the model thinks is happening and what is actually happening opens third.

Business impact shows up last.

Most teams only watch business metrics, which means they are always weeks late.

Went deep on this on YouTube (TagAlongWithVarun), covering specific metrics, the order they fire in, why the order matters, and how to think about building this monitoring layer: https://www.youtube.com/watch?v=ZK3zK8flydo

What silent model failures have you run into?

u/Extension_Key_5970 — 27 days ago
▲ 23 r/mlops

what ML topics do you actually want explained from a production/infra angle?

I have been making some YouTube videos around MLOps and ML platform stuff, which is system design, real pipelines on k8s, that kind of thing. The channel is still small, and I am figuring out what to focus on next.

Rather than guessing, figured I would just ask here since this sub is basically my target audience. What are you running into at work or in interviews that you wish had a clear walkthrough?

Not the "here is the theory" kind, more like "here is what it actually looks like in production."

Some stuff I have already covered: ETL pipelines, ML training pipelines on airflow/mlflow, inference system design, and RAG architecture; open to anything, though.

No wrong answers, just tell me what would actually be useful.

Edit 1: If anyone wants to look into my current videos: https://www.youtube.com/@TagAlongWithVarun

u/Extension_Key_5970 — 1 month ago
▲ 9 r/mlops

The side of RAG that most tutorials skip, what actually runs behind the scenes, useful for system design prep too

Most RAG content focuses on LangChain, prompt templates, chunking. Almost none of it covers what the infrastructure looks like when real users are hitting it concurrently.

What actually runs when someone asks “what is our parental leave policy” to an internal doc search:

Not every query needs RAG. “Hi” does not need to search 10,000 documents. A classifier at the front catches these. Semantic cache handles repeated questions. Saves a lot of unnecessary GPU spend.

Pure vector search misses exact terms like “Policy HR-2024-37.” Embeddings capture meaning, not identifiers. Most production setups run semantic and keyword search in parallel, merge results, and rerank. Search finds similar docs. Reranking asks “does this actually answer the question.” Different problem entirely.

Only the LLM needs GPU. Router, embedding service, search, reranker may ran on CPU. Each scales independently. Matters for cost and K8s design.

“Server is up” is not the same as “answers are good.” You need retrieval quality, hallucination rate, per-component latency. Hallucination detection is still unsolved. Best you can do is automated checks plus source links.

Chunking makes or breaks retrieval. Too small, no context. Too large, noise. 200 to 500 tokens with overlap, adjust based on document structure.

Self-host vs API depends on privacy, volume, budget. No universal answer.

If anyone wants to see how these components actually connect, I walked through the request flow visually on YouTube: https://youtu.be/u48D4VVHBKM.

How are you handling retrieval? Hybrid or vector-only? Keen to know more.

u/Extension_Key_5970 — 1 month ago
▲ 0 r/mlops

What I got wrong about MLOps interviews (coming from infrastructure)

Kept walking into interviews talking about Kubernetes, GPU scheduling, CI/CD. All real experience. The interviewer would listen, then ask, "How would you detect a data distribution shift?" and I had nothing.

Not because I was not good enough. But because I was talking about my world, not theirs.

The shift that helped: reframing what I already know.

Before: "I set up autoscaling using HPA targeting 70% GPU utilization."

After: "I reduced serving latency from 200ms to 45ms by autoscaling on request queue depth, which improved real-time prediction experience."

Same work. The second version connects to what ML teams care about.

Another thing that tripped me up: MLOps means different things at different companies. Some roles are 90% infra. Some are hybrid. Some are 90% ML-focused — drift detection, retraining, A/B testing. If the interview feels misaligned, it is probably a mismatch in the spectrum, not your skills.

Talked about this more in a recent video: https://www.youtube.com/watch?v=oWSXO3a7QSs&t=22s, what to focus on, what I wish I knew earlier.

u/Extension_Key_5970 — 1 month ago
▲ 32 r/mlops

Built a production ETL pipeline on Kubernetes for MLOps End to End series, sharing the architecture and design thinking

Before training any model, you need clean data. Sounds obvious, but most MLOps content skips straight to the model. So I started my series with what actually comes first, a data pipeline.

Built a complete ETL pipeline that pulls real crypto market data from the Binance API and loads it into PostgreSQL. 2.28 million rows of structured OHLCV data. Runs on Kubernetes with Airflow using KubernetesExecutor.

Some design decisions that might be useful:

  • Why Store data twice?
  • Why KubernetesExecutor over CeleryExecutor?
  • Why Parallel extraction?
  • Why Pre-flight checks before ETL starts?
  • Why Connection pooling for bulk loads?

Whole infra deploys with one command. Docker Compose option included for people without a K8s cluster.

I've also recorded a full live walkthrough on https://youtu.be/5HBeVZ7uMlg if you'd like to see it running end to end.

And of course for Patience Readers: Medium

Please let me know future topics where you want something related to ML Production scenarios

u/Extension_Key_5970 — 2 months ago
▲ 56 r/mlops

How I approach MLOps system design questions in interviews: sharing the thinking, not just the diagram

Got asked "design a data ingestion pipeline for an ML team that needs daily data from 3 external APIs" in a system design round.

Sharing my approach.

Ask clarifying questions first. Most candidates skip this and start drawing immediately. But every answer below changes the design:

  • JSON vs streaming vs flat files? Changes the entire ingestion layer.
  • 5 GB/day vs 50 GB vs 1 TB? Python + PostgreSQL vs Spark vs full data lake with Delta Lake/Iceberg.
  • Real-time vs daily batch? Kafka + Flink vs a scheduled Airflow DAG. Massive complexity difference.
  • One team vs twenty? Simple DB vs access control, data catalogue, feature store.

I assumed: structured JSON, 5-10 GB/day, daily batch, single team, Kubernetes available.

The pipeline:

3 API sources → Airflow (KubernetesExecutor, one pod per task) → parallel extraction → raw JSON stored in MinIO untouched → transform (clean, cast, validate) → PostgreSQL.

Key pattern: store raw and processed separately. Transform logic has a bug? Fix code, reprocess from raw. No re-fetching from APIs. Interviewer asks, "Reprocess last month?" --> You have an answer.

Production concerns that matter:

  • Exponential backoff on retries (1 min, 5 min, 15 min)
  • Idempotency: re-running the same date must not create duplicates (upsert, partition overwrite, or staging table merge)
  • Data quality checks after every load — null counts, row counts, duplicates
  • Backfill support from raw storage

Mistakes I have seen (and made):

  • Saying "I would use Kafka" before knowing volume or freshness
  • No raw storage layer = no reprocessing ability
  • Only describing the happy path, never mentioning failures
  • Over-engineering a single-team problem with Spark Streaming and data mesh

Actually built this pipeline on Kubernetes with real Binance API data. Code: github.com/var1914/mlops-boilerplate

Full visual walkthrough on YouTube

u/Extension_Key_5970 — 2 months ago