r/mlops

I am looking for

How a devops guy trains model?
How a devops guy creates pipeline for the model?
How a devops guy deploys model?
How a devops guy monitors model?
Also, best practices of MLOps.

Something that rarely shows up in MLOps content but is a real part of the job: when an org decides to self-host an LLM, someone needs to figure out which model to actually run and whether the cost makes sense.

It is not "pick the highest benchmark score." It is "what fits on the hardware we can afford and still meets the quality bar."

Example of the tradeoff: Qwen3.5-35B-A3B has 35 billion total parameters but only 3 billion active (MoE). Fits on a single H100. On many tasks it beats Haiku-class API models. Cost: one GPU instance.

GLM 5.2 has 745 billion parameters, about 40 billion active. Open weights, MIT license. Benchmarks within 1 to 3 points of Sonnet 4.6 on coding tasks. But it needs roughly 744 to 890 GB VRAM in FP8. That is 10 to 12 H100s minimum.

So the question becomes: is that marginal quality bump worth 10x the GPU cost? For most use cases, no. For some, yes. That is the analysis.

Tools like AIPerf (github.com/ai-dynamo/aiperf) help with this. You benchmark TTFT, inter-token latency, throughput at different concurrency levels, on your actual hardware, with your actual prompts. Public benchmarks tell you what a model can do in ideal conditions. AIPerf tells you what it does on your infra.

The thinking behind model selection:

• Define what "good enough" means for the use case. Internal doc QA does not need frontier reasoning.
• Benchmark on your workload, not public leaderboards.
• Calculate cost per request, not cost per token. Factor in throughput and concurrency.
• Consider operational overhead. One GPU is simple. A 12-GPU cluster with tensor parallelism is not.

This is the kind of work that makes MLOps different from just knowing tools. Understanding model architectures (dense vs MoE, active vs total parameters), GPU memory math, and cost-quality tradeoffs is part of the job.

Has anyone here gone through a model selection process for internal hosting? Curious what you ended up picking and why.

I’m currently pursuing my BS in Remote Sensing and GIS. For those outside the field, it’s basically the science of dealing with massive satellite data, geographic coordinate systems, and mapping the Earth.

​The problem? I’ve realized my true passion isn’t traditional map-making or static analysis—I want to be an MLOps Engineer.

​I've been teaching myself Web Development (HTML/CSS/JS) to build interfaces, and I've dived deep into machine learning infrastructure, model packaging, and deployment automation.

​Every time I look at junior MLOps job postings, they ask for general software engineering or DevOps backgrounds. I feel like my degree title is going to get my resume thrown out instantly by recruiters who think I just make maps in ArcGIS all day.

​But here is my theory: satellite imagery is notoriously painful to work with (multi-gigabyte files, multi-spectral bands, crazy data pipelines). It feels like the perfect playground for robust MLOps engineering.

​I want some brutal honesty from the community:

​Am I fighting an uphill battle? How hard is it to land an MLOps role when your formal background is in earth science/GIS instead of pure Computer Science?

​How do I bridge the gap in my portfolio? Right now, I'm planning a project where I build a web app dashboard that automatically pulls open-source satellite data, runs it through a containerized computer vision model via a Docker/CI/CD pipeline, and renders the predictions interactively on a map. Is that enough to prove I can handle production-grade infrastructure?

​Has anyone successfully transitioned from a specialized domain (like GIS/Remote Sensing) into core ML/Platform engineering? What was the turning point on your resume?

​Thanks in advance. Feeling a bit stuck between what my degree says and what I actually want to build.

Trying to understand how teams handle this in practice.

When you deploy a self-hosted open-weight model for an agent (something that makes a bunch of tool calls in a row), who decides it’s ready to go live and what does that check actually look like?

From what I’ve seen, the usual benchmark scores don’t really predict whether a model holds up over a long multi-step run. It can look fine and then fail quietly once it’s live. And the same model behaves differently depending on the runtime and quantization it’s served with, so “passed in testing” and “works in prod” aren’t the same thing.

For people running this for real:

• Is there a real pre-deployment check for self-hosted models, or is it mostly deploy-and-monitor?

• Who owns that gate the ML team, platform/ops, or nobody clearly?

• What do you wish you’d caught before it went live instead of after?

Trying to learn how this works in the real world. What’s your setup?

I’ve been interviewing for several Staff/Principal Platform Engineering and MLOps roles around Silicon Valley recently, and I’ve noticed an interesting pattern. Curious if others are seeing the same thing.

The job titles often sound like:

AI Platform Engineer
ML Platform Engineer
Platform Architect
Platform Engineering
Infrastructure Architect
MLOps Engineer

But once the technical interview starts, the discussion quickly narrows into Kubernetes operations.

Typical probing topics include:

Kubernetes scheduling internals
Pod lifecycle and failure scenarios
CNI/CSI details
ArgoCD deployment mechanics
Helm charts
Terraform modules and remote state
GitOps workflows
EKS/GKE operational issues
Container networking
Service mesh

Production debugging:
On-call incidents
Disk pressure / memory pressure
etcd behavior
Rolling upgrades

Very little time is spent discussing:
ML platform architecture
Feature stores
Model lifecycle
Training infrastructure
Batch vs streaming ML pipelines
Data contracts
AI infrastructure strategy
Platform architecture tradeoffs
Multi-team platform evolution

Instead, many interviews feel like they’re looking for someone who has spent years running production Kubernetes clusters and handling operational toil.
One hiring manager described the role as “Platform Engineering,” but nearly every technical question centered around daily Kubernetes operations, CI/CD mechanics, production troubleshooting, and infrastructure automation.

Compensation wasn’t low either. These were generally Staff-level roles in the Sunnyvale/Santa Clara area with base salaries around $220k–260k+, which surprised me because I expected more architectural discussion at that level. But Hiring manger puts a hard line of $200k max all inclusive. The JD compensation is fake to attract staff level candidates but pay was mid-level SRE $160-180k + bonus if any .

My impression is that many companies are using “Platform,” “AI Platform,” or “MLOps” as umbrella titles for what is fundamentally senior Kubernetes platform operations.

I’m not saying that’s wrong—someone has to build and operate reliable infrastructure—but the title and interview focus often don’t match.

Curious what others are seeing.

Questions for the community:
Are “Platform Engineering” and “MLOps” titles increasingly becoming Kubernetes operations roles?
How much architecture discussion do you typically see in Staff/Principal interviews?
Are companies intentionally broadening titles to attract candidates, or has the definition of platform engineering genuinely shifted toward infrastructure operations?
For those hiring Staff engineers, what percentage of the interview is architecture versus deep operational troubleshooting?
Interested to hear experiences from both hiring managers and candidates.

Disclosures: Used AI assistance to streamline my thoughts for better narrative and grammar.

There is a lot of hidden language. Once you decode it, the noise drops.

Recruiter/HM Translation Table

What They Say |What It Often Means
“Fast-moving process” |Urgent backfill / overloaded team
“Hands-on from day one” |No ramp, cleanup work
“Platform engineer” |Could mean operator, not owner
“AIOps/AI automation” |Ops backlog with AI wrapper
“Flexible on level” |They may down-level based on price
“Competitive compensation” |Budget may be capped
“Looking for right fit” |Strategic role, slower timeline
“High ownership” |Could mean high accountability, low support
“Wear many hats” |Understaffed
“Support internal customers” |Service posture
“Modernization” |Could mean real roadmap or cleanup debt
“Startup mindset” in big company |Do more with less
“Can you jump on a call today?” |Urgency, not necessarily quality
“What are you looking for?” |They are testing price, scope, urgency
“Are you hands-on?” |Can you execute tickets/scripts/on-call?
“5+ years required” for senior title |Mid-level budget
“We’re still defining the role” |Scope risk / fishing expedition
“Strong communication” |Cross-team friction likely My Mental Shortcut: Hype words do not matter. Ownership, authority, budget, and respect matter.

Ignore title inflation:

• Platform
• AI
• AIOps
• Automation
• Modernization
• Transformation
• Observability
• Cloud-native

Ask what the person actually owns.

One question cuts through most noise: “What will this person own after the first 90 days: roadmap, architecture decisions, or operational coverage?”

If they cannot answer, it is noise.

Second question: “Is this new headcount for a strategic initiative, or a backfill for an overloaded team?”

That separates opportunity from cleanup. Walk away - market survey, exploratory interviews, low-level job.

Once you are reading the market correctly. The next step is protecting your calendar.

Disclosures: Used AI to format the interview language for better communication and sharing

I build and ship AI products for a living, and I just gave a talk on the thing nobody puts in the demo: what happens to your LLM app after it's live and real users start doing weird stuff to it.

The pattern I keep seeing: a feature demos perfectly, ships, and then quietly degrades for weeks before anyone notices — because there's no instrumentation to catch it. The model didn't "break," it just started doing something slightly wrong some percentage of the time and nobody was watching that slice.

The three things I argued you actually need:

Evals you run continuously, not once. Most teams treat evals like a pre-launch checkbox. The useful version is a regression suite that runs against real traffic samples so you catch drift before users report it.

LLM-as-judge, but with a sanity check. It scales review way past what a human team can do, but it's not free — you have to validate the judge against human labels periodically, or you're just trusting one black box to grade another.

A real failure-case library. Every prod incident becomes a permanent test case. This is the boring part that actually compounds.

Curious how others here handle this — specifically: do you trust LLM-as-judge in your pipeline, or have you been burned by it? My stack leans on Langfuse for tracing, Portkey as the gateway, and Sentry for the app layer, but I'm always looking for what's working for people.

Once you're calling more than one LLM provider, you eventually hit the question of whether to put a gateway (or router, same idea, the terms get used interchangeably) in front of them. I resisted this for a while, it felt like premature infrastructure, until a provider had a rough hour, our app just sat there serving 500s with no fallback path, and support tickets piled up for most of a day. That's usually the moment teams actually go looking. Here's how the main options compare in 2026.

First, what an AI gateway actually does, so the comparison makes sense. It sits between your app and the providers, gives you one endpoint for many models, lets you swap models without code changes, handles fallbacks when a provider fails, and centralizes cost tracking and budget limits. Some go further into governance and org-wide routing policy. The differences below are mostly about how far past "one endpoint for many models" each one goes.

OpenRouter

The easiest possible on-ramp. You point your existing OpenAI client at their endpoint and it basically works, plus you get access to a huge catalog of models, more than you'll probably ever use. Pricing is usage-based with a small markup on top of provider costs, or bring your own keys. For getting started, prototyping, or hobby work, it's hard to beat on speed-to-first-call.

Where it stops short is the control plane. It's more "here's access to every model" than "here's how my org governs LLM usage." Budget caps per team, fine-grained routing policy, governance, those aren't its focus. Fallback exists but it's defined per-request rather than as an org-wide policy.

Best for: fast model access, prototyping, broad model coverage.

LiteLLM

The open-source workhorse, and probably the most widely deployed thing on this list in practice. If you want a self-hosted, code-level abstraction across providers, LiteLLM is the one most teams reach for first. It normalizes the API across 100+ providers so you're not maintaining provider-specific request formats, runs as a proxy or a library, and has an active community and fast provider support, new models tend to show up quickly. The proxy also does key management, budgets, and rate limits, more than people assume it can. It's free as open source, with a paid enterprise tier for SSO, audit logs, and support.

The tradeoff is the usual open-source one, you own it. You're operating the proxy, handling upgrades, and building your own dashboards and higher-level governance around it. Great if you want control and have the ops capacity, more work if you don't, and that maintenance cost is easy to underestimate on day one.

Best for: teams wanting an open-source, self-hosted abstraction with full control and broad provider support.

Portkey

A real step up into managed production controls. Portkey has proper request-level reliability, configurable retries and fallbacks (you set conditions on what counts as a failure and where it routes next), semantic caching that genuinely cuts repeat-query cost, guardrails, and solid observability. Pricing is usage-based with a free tier to start, which lowers the barrier to trying it. If your main pain is per-request reliability and cost on a service or two, it's mature and well-built, and its failure-condition config is the most granular here.

Where it felt limited to me was at the org level. When you've got many services and want one routing policy applied everywhere, it leans more toward per-request configuration than "define the rule once, apply it across the whole organization."

Best for: teams that need strong request-level reliability, granular fallback control, and caching.

Orq.ai

Orq's gateway (you'll still see it called the Router, the rename to Gateway is recent so both terms are floating around) is the most policy-oriented of the group. Budget controls and fallback chains are defined centrally, so when a provider degrades, the fallback applies across every service routing through it without per-app changes. If you're running several apps or teams through one layer and want consistent routing policy, governance, and cost control in one place, that's the angle it's built for.

The bigger differentiator is that the gateway isn't a standalone product, it's part of a full LLMOps platform, so routing sits next to prompt management, evals, and observability rather than being a separate tool you reconcile with everything else. It's managed, and the compliance story (SOC2, GDPR, EU AI Act, EU-based company) is real if you're in a regulated space.

The honest tradeoffs: it's a broader platform, so if all you want is a thin routing layer and nothing else, it's more than you need, and Portkey gives you more granular per-request failure knobs. It's also a newer, smaller player than the incumbents, so the community and battle-tested-at-extreme-scale factor isn't the same as something like LiteLLM yet. Worth weighing if you're purely optimizing for a minimal gateway rather than a platform.

Best for: teams wanting centralized routing policy across multiple apps, or a gateway that's part of a broader lifecycle platform.

The short version

If you just want model access fast, OpenRouter. If you want open source and you're happy owning it, LiteLLM, which is also the safest bet for broad provider coverage and community support. If your pain is granular request-level reliability and caching on a few services, Portkey. If you're routing multiple apps through one layer and want centralized policy plus prompt management, evals, and observability in the same place rather than a standalone gateway, that's where Orq.ai fits best.

The real fork is whether your problem is "give me models" or "help me govern how my org uses them." The first points to OpenRouter or LiteLLM, the second to Portkey or Orq, and which of those two depends on whether you're solving reliability on a few services or policy across many. Pick based on the problem you actually have today, not the one you might have at 10x scale, you can always move up the list when you get there.

For the first 4 months of running LLMs in production, our observability was mostly vibes. App responded → good. App timed out → bad, we basically had no insight into what was actually happening.
Soon we realised setting up proper LLM observability was one of the highest-roi things we did. Here's what I learned setting up this,

So, our setup is primarily, a gateway-level tracing with OpenTelemetry export to our existing stack. Every LLM request now has: model called, input tokens, output tokens, latency (wall time + time-to-first-token + inter-token latency), cost, and metadata tags for user/team/feature/environment. These traces land in our Datadog alongside regular app traces.
For agent workflows, we capture the full trace: which tools were called, in what order, what the model's intermediate reasoning was (for models that support it), final response. This is invaluable for debugging.

A few really important insights this revealed for us,

1. One team was calling GPT-4o for a task that needed basic extraction. Model was 10x more expensive than needed and actually slower, so we moved to a smaller model, cost dropped massively to about 85% for that feature
2. We had a semantic cache enabled but with the wrong similarity threshold. Half our "cache hits" were on requests that shouldn't have matched, tuned this and improved cache hit rate meaningfully
3. One of our RAG pipelines had an embedding call that was adding 400ms every time. Not obvious without per-step latency, fixed it by caching the embeddings.
4. Our "prod" and "dev" environments were sharing rate limit quotas. Dev was sometimes throttling prod - added environment-based quota separation.

We use truefoundry's gateway for the OTEL export and it pipes directly into our existing datadog setup without a separate observability vendor. But honestly the specific tool matters less than the pattern: get your LLM traces into whatever stack your team already uses for the rest of your services. The value is in correlation, not in a standalone llm dashboard nobody checks

The single question worth asking right now is if someone asked you why your p95 llm latency spiked on a random Tuesday, could you answer it?

If not, that's the gap, what's everyone else exporting to? are you guys using datadog/grafana or using dedicated llm observability tools like langfuse or arize?

LLM governance is one of those terms that gets used constantly these days, but with wildly different meanings depending on who's talking. I've heard it mean everything from we have a content filter to we have a full compliance program with audit trails and model risk management

We've been building this out for the past year. So, here's how I'd now break it down into layers from my understanding that actually map to things you build:

Layer 1: Access control who can call which models, with what keys, with what limits. This is the gateway layer - virtual keys, per-team rate limits, budget caps. Most teams start here because cost incidents force it.

Layer 2: Content governance what can go in and come out. Input guardrails (prompt injection detection, PII scrubbing before data leaves your perimeter), output guardrails (content moderation, safety checks). The key design question: validate-only (flag and block) vs mutate (modify the content). Both are useful for different cases.

Layer 3: Audit and observability every request logged with enough context to answer: who made it, what model, what it cost, what the prompt contained, what the response was. The hard part isn't capturing the data - it's making it queryable in a format a compliance team can actually use, not raw JSON logs.

Layer 4: Model risk management which models are approved for which use cases. Who decides when a new model goes on the approved list. What happens when a model is deprecated. This is the most organization-specific layer and usually the last one teams formalize.

Layer 5: Agent and tool governance if you're running agents that call tools via MCP: which agents can call which tools, under which user identity, with what audit trail per invocation. This layer didn't exist two years ago and most governance frameworks haven't caught up to it.

Most teams I've talked to have layer 1 in some form, maybe layer 2, and are improvising on 3-5... what layer is causing the most pain in your current setup?

I’ve spent 30 years in hardware operations and NPI at places like HPE. I built the GPU Compute Index because I noticed everyone was buying GPUs based on 'sticker price' while ignoring the physics of interconnects.

If you're training large models (70B+) on standard 100GbE Ethernet, you're likely paying a 40% hidden tax because your GPUs are sitting idle waiting for gradients. I put this calculator online for free so the community can run real TCO models.

There's also a 36-month 'Build vs. Rent' logic included. Hope this helps some of you save on your cloud bills.

​

So i run evals for a multi-agent system at work and right now my main approach is LLM-as-a-judge against a gold set, plus some semantic similarity scoring. And honestly... it works until it doesn't.

The judge is inconsistent. Same output, slightly different prompt phrasing, different verdict. It's biased toward longer answers, it rationalizes things the gold set clearly says are wrong, and calibrating it feels like im just stacking prompt rules on top of prompt rules hoping the false positives go down. Which they do, partially, but I don't fully trust the number at the end.

What I'm trying to figure out:

- do you treat LLM-as-judge as a real signal or just a smoke test before human review

- how do you handle judge drift when you swap the underlying model

- for agent systems specifically, are you scoring final output or the whole trajectory? feels like scoring just the end misses a lot

- anyone actually getting value out of semantic similarity or is it mostly noise

Not looking for a vendor pitch, genuinely want to know what's working for people running this stuff day to day. Feels like everyone has a different homegrown setup and nobody's sure theirs is good.

Hello MLOps folks!

​I would like to bring up an interesting topic that I am highly interested in. It is clear that we are now facing the next frontier of AI applied to the real world: Physical AI (robotics).

​I am looking for fresh ideas or insights from experienced people working in robotics, whether from the perspective of a researcher/roboticist or an MLOps/infrastructure engineer. Specifically, I want to discuss the different setups and platforms robotics companies are using to scale their experimentation and training, and how they are navigating this emerging sector.

​I would love to hear about the architectures you are using or how you would design them. Are you using Kubernetes, services like AWS Batch, or frameworks like Ray? What about tracking tools like Weights & Biases or MLflow?

​Robotics comes with major challenges, such as non-deterministic outcomes (similar to LLMs) and the sim-to-real gap. This means that things that work in simulation must behave the same way on a physical robot.

- ​How do you handle these scenarios?

- ​What quality gates do you use to ensure safety and accuracy?

- ​How do you manage different training pipelines for various research phases, such as teacher-student distillation or running Hyperparameter Optimization (HPO) on just a single phase?

​Happy to discuss!

We have been on Airflow for about 2 years now (350 DAG, team of 6 data engineers). The scheduler keeps choking, DAG parsing takes forever when someone pushes a change and honeslty maintenaing the infra around it eats more time than writing actual pipelines.

I have looked at Dagster n Perfect but bot still feel very python centric which is part of what's burning us out. Aynone moved to sth fundamentally different ?

Hi everyone,
I’m currently trying to transition into ML infrastructure (or ML platform engineering, as many companies call it these days).
My background is primarily in DevOps, cloud infrastructure, and release engineering. I’ve worked extensively with Kubernetes, spent some time at VMware Tanzu, and have mostly used AWS, although I have experience across other cloud providers as well.
More recently, I completed a Master’s in AI, so I have a solid understanding of modern LLMs and multimodal models from the model side. What I feel I’m missing is hands-on experience with production ML systems.
I’m currently trying to understand ML workload scheduling and orchestration. I see that many organizations build these workloads on Kubernetes, but there seems to be a growing ecosystem of tools, and I’m having trouble understanding what has become the industry standard.
Some of the projects I’ve come across are:
Kubeflow
Kueue
KubeRay
Volcano
Argo
Flyte
Airflow (in some cases)
I realize many of these tools solve different problems and are often used together, but I’d love to understand how they fit into a modern ML platform.
For example, what does a typical production ML training/inference pipeline look like today (excluding model serving engines like vLLM or other LLM-specific runtimes)? I’m more interested in the general platform architecture and how training jobs are scheduled, orchestrated, tracked, and deployed.
Also, are there any tools that you would consider “must know” for someone aiming for ML infrastructure/platform engineering roles? Is there anything that has effectively become the de facto standard in the industry?
Finally, do you think any certifications are actually valuable for breaking into this field, or is it better to focus on building projects and gaining hands-on experience?
Thanks in advance! I’d really appreciate hearing from people working in ML platform engineering or MLOps today.

Guys, as I've mentioned in other posts, I want to be a machine learning engineer. We already have the production model implemented. The idea is to monitor it and, if it degrades, create a training pipeline to manage the entire manual process, from loading new data to retraining, validation, automated deployment, and so on. I've already done this with Vertex AI Pipelines, but it's a paid tool and my credits have expired. Since I want to gain experience with a real production process, what free or open-source tool should I start with for monitoring and pipelines as a beginner? I've done some research and there are too many tools (ZenML, Kubleflow, etc.). I'm lost; I don't know which one to choose or which one a company would require.