r/mlops

eight months in and a fraud detection system we were brought onto started giving results that just felt wrong. nothing was crashing nothing was throwing errors. it was more like.. the outputs were off in this subtle way that the team kept second guessing themselves about. client had already spent a good chunk of time internally trying to figure it out before they came to us. model itself was honestly fine. solid work. they were 100% sure we were coming in to blow it up and rebuild from scratch.

we didnt touch the model at all actually. what we kept finding was that everything around it was basically missing. no drift detection no retraining setup no versioned datasets just nothing. the thing had been running against a data distribution that was already like six months old. real world patterns had moved on and the model was still answering for a world that didnt really exist anymore. and because nothing was monitoring any of this the dashboards just kept showing green. that part made it genuinely tricky to even convince people something was wrong.

we put the MLOps layer together from scratch. drift based retraining triggers Evidently AI for monitoring shadow testing before anything got promoted proper dataset versioning the whole thing. took about six weeks and by the end the outputs were where they needed to be infra costs dropped 34% and the model that was already there started working the way it was built to. never retrained it never rebuilt it. just finally gave it the infrastructure it needed around it.

Open source maintainers are currently acting as unpaid QA for poorly prompted LLM scripts. If you manage a repository with any decent footprint right now, you already know the metrics. The ratio of human-written code to automated garbage has inverted.

I checked the logs across three of our infrastructure repos yesterday. Over the last 30 days, we saw a massive spike in automated pull requests. These are not helpful dependency updates. They are looping scripts tied to agent frameworks, submitting circular logic fixes, hallucinated bug bounties, and unprompted refactors that break build pipelines. The volume is high enough that it actively costs compute money in CI runs.

Standard rate limiting does not work here. GitHub's native tools are lagging behind the volume. A lot of teams are trying to implement complex heuristic checks or relying on third-party bot blockers. We found a much simpler, deterministic fix.

We stopped the spam entirely using Git's native --author flag.

Here is the data on what is actually happening and how to implement the block at the repository level.

The anatomy of agent spam is predictable. When a developer uses an agent like CC or a local script to scrape and push, the Git client constructs a commit object. A standard Git commit object contains a tree, a parent, an author, a committer, and the message. The critical failure point for most automated AI tools is that they do not natively handle Git identity management well. They default to the environment variables of the host machine or use hardcoded placeholder strings generated by the LLM framework itself.

If you run a local LLM or an API-driven agent, the scripts executing the Git commands often leave a fingerprint in the --author string. Sometimes it is explicit, like 'Author: AI Agent <bot@example.com>'. More often, it is a mismatch between the authenticated GitHub user pushing the code and the internal Git author email attached to the commit hash.

We set up a pre-receive hook in our enterprise environment and a simple GitHub Action for our public repos to enforce strict author validation. The logic is basic but effective.

When a push event triggers, the pipeline checks the commit history. It extracts the author string using `git log -1 --format='%an <%ae>'`. We then validate this against a strict allowlist of email domains for internal contributors, or for public repos, we enforce a strict requirement that the Git author matches the GitHub actor pushing the branch, alongside cryptographic signature verification.

Agents fail this check instantly.

Most script-kiddie setups deploying autonomous coding agents do not bother to configure GPG signing. They do not ensure the Git author matches the GitHub API token identity. They just run a blind commit and pipe it to a push loop.

I ran the numbers on a honeypot repository we maintain just to track this behavior. In a 14-day window, the repo received 412 automated pull requests. I implemented the author validation check. 412 out of 412 agent-driven PRs failed the pre-check and were auto-closed before triggering any heavy CI workflows. That is a 100 percent drop in compute waste. The false positive rate for legitimate human contributors was near zero, provided they had actually configured their local Git environment correctly, which is a baseline expectation anyway.

Let's look at the mechanics of how the --author flag actually operates in this context. Git separates the concept of the Author and the Committer. When an AI agent generates code, the script executing the commands will often attempt to spoof or manipulate these fields. By enforcing a strict parsing of the --author parameter in your CI/CD pipeline, you trap the bots. We use a GitHub Action that runs a diff against the HEAD. If the email domain belongs to a known ephemeral email provider, a local non-routable address like .local, or a generic string often hardcoded by popular agent libraries, the pipeline exits with a non-zero status.

We started aggressively filtering based on the discrepancy between the GitHub Actor making the API request and the parsed Git Author. AI scripts are notoriously bad at state management across different authentication layers. The bot account pushing the code almost never matches the internal Git config of the container that generated the code.

This mismatch is the exploit.

Consider the actual cost here. Every time a bot opens a PR, GitHub Actions provisions a runner. If you have a decent test suite, that runner might spin up database containers, compile code, and run tests. Let's assume a conservative cost of 20 cents per run in compute time. If your repo gets hit by 500 bot PRs a month, that is $100 burned. For enterprise teams managing hundreds of repos, this easily scales into thousands of dollars of wasted infrastructure spend simply because someone hooked an open-source LLM to the GitHub API.

I refuse to pay for someone else's badly prompted experiment.

The implementation is straightforward. You do not need to buy a third-party security product. You write a bash script. Block unverified commits. Add a step in your primary workflow file that validates the commit author. Reject any push where the committer email is not tied to a verified human domain or an explicitly allowed internal service account.

Tested on prod. The drop in noise is immediate.

The industry is going to have to standardize around authenticated machine identities soon. Until platforms introduce a dedicated bot flag at the push layer, repository maintainers have to defend their own infrastructure. Rely on the cryptographic and structural metadata of the version control system itself.

Check your repository analytics. Look at the ratio of closed PRs to merged PRs over the last 90 days. If that number is trending upward, you have a bot problem. Apply the filter. Benchmark the results.

Numbers don't lie. How are you handling the automated sludge right now? Are you manually closing these tickets, or have you automated the rejection pipeline?

I am devops and cloud engineer with 3 years of experience and a good background in software engineering and backend development using languages like python, typescript, and java. I have a CS degree and good background on math and computer science subjects and I'm considering switching to mlops role. I already work with the cloud, CI pipelines, and kubernetes clusters and infra everyday so I think I already cover a good portion of the requirements for the job but I am pretty clueless to anything related to machine learning and how models lifecycles are managed, integrated like in SDLC. I was wondering about the amount of new materials I would need to study, technologies and concepts I will need to get familiar with and hands on experience required. If someone here came from a similar background, do you consider the transition from devops to mlops an easy or even viable medium-term goal? Is there any materials do you recommend specifically for engineers with existing technical background in software and cloud?

Hi everyone, I’m currently a full-stack developer, but I’m thinking about transitioning to an MLOps engineer role. I’m curious about the day-to-day work in MLOps and whether it’s a safe career move for the future. I’m concerned because I struggle with math, and I know it’s important in ML. So, is it a good idea to make this switch? Is MLOps a future-proof, AI-proof kind of path? I’d really appreciate your insights!

Hi all, I wanted to drop my website in this subreddit as a way to publicize my work and services.

I'm offering consulting services to offer a post-training step that optimizes models for deployment by avoiding expensive inference, feature lookup, executing kernels, etc. when not necessary.

The underlying tech is essentially advanced analytics that cross-correlates high dimensional data with predictions, and finds regions of the data space that don't require compute-heavy resources.

I can help with systems where the objective is to reduce cost, increase throughput, reduce latency, and reduce energy usage.

If you're interested in a pilot or have questions, please do reach out here or book a meeting through the website. I love working on technical problems so I'm very committed to solving yours.

https://compressmodels.github.io

We are a small team working in new ML project, and we are evaluating different orchestration tools like Trigger.dev, Prefect, Temporal, and others. However, before making sure that whatever tool we chose would meet our needs, we must ensure that changing the tool in the case of it being unfit for our work would not turn into a problem. I feel like there is no winning here because once the commitment is made, there is little to do about it.

Your opinion in the matter would be much appreciated:

have you had an experience of having to change orchestration tools mid-project? what made you do so?
why did you think it was necessary to choose that particular orchestration tool?
is there any set of conditions for those choices, or everything depends on the particular circumstances?

For teams deploying LLM/agentic systems into financial workflows, how real is the operational recovery/problem-management side once these systems start taking actions instead of just generating text?

I’m especially curious about cases where the workflow technically “succeeds” at first, but becomes wrong later because of reconciliation mismatches, stale context, invalid state transitions, settlement issues, etc.

Are teams actually defining explicit correctness boundaries/checkpoints/reversibility ahead of deployment, or is most recovery still manual investigation after something breaks?

Trying to understand how mature this is in practice.

Figure AI just ran their F.03 units for over 30 hours straight. The livestream was raw. No cuts. Three units—Bob, Frank, and Gary—cleared 28,000 packages at the 24-hour mark and kept moving well past 30. Forget the emotional narrative about human replacements. Let us look at the edge compute, thermal management, and actuator degradation data. Numbers do not lie.

When you push a bipedal robot to operate for 30 continuous hours, you are no longer doing a robotics demo. You are doing an endurance benchmark for edge MLOps. The F.03 runs on the Helix-02 system. In order to sort 28,000 packages over a day, the vision models and motion planning algorithms are executing millions of forward passes. If they offloaded this to the cloud, the network latency jitter would inevitably cause a dropped package or a collision. A 200-millisecond lag spike means the robot misses the conveyor timing. The fact that they operated unsupervised for this duration proves the inference is fully localized and quantized to run within the thermal limits of the chassis.

Let us look deeper into the inference latency. To run a bipedal robot, you are typically running a multi-modal transformer for high-level reasoning and a rapid control policy for lower-level kinematics. If the vision model is operating at 30 frames per second, that is 108,000 inferences per hour. Over a 30-hour shift, each robot is processing over 3.2 million visual frames. You cannot stream that to an endpoint. The VRAM constraints on the local edge hardware must be incredibly tight. They are likely running a heavily distilled architecture purely for the vision-action mapping. The control loop needs to run at something like 500Hz to maintain balance and precision during the package sorting.

Let us talk about thermal throttling. Continuous operation means the battery discharge rate and the compute package are generating heat that has nowhere to go but out through the passive casing. To run 30 hours without a localized shutdown means the inference budget is ruthlessly optimized. They are likely using aggressive dynamic voltage scaling. I ran the numbers on standard industrial arm power draw versus compute overhead. For a humanoid to stay active this long, the physical movements must be heavily reliant on energy recovery from the actuators during deceleration phases, paired with a low-power standby state for the inference chips between grabs.

The mechanical benchmark is equally severe. Figure’s BotQ facility in California is now producing one F.03 unit per hour. That is a 24x increase in throughput in just 120 days. They have shipped over 350 units and built more than 9,000 actuators. This scale matters because of the failure rates. At 28,000 packages handled by three robots, we are looking at roughly 9,333 sort cycles per robot in the first 24 hours alone. Each cycle requires multi-axis coordination. Shoulders, elbows, wrists, and the tactile grippers are all firing. A standard industrial actuator starts showing thermal drift after a few hours of continuous cyclic loading. The F.03 actuators sustained 30 hours of continuous load without requiring a manual recalibration. We saw another data point where seven units ran autonomous self-calibration and stress-testing for 90 minutes straight. They are essentially running localized closed-loop tuning on their own hardware while operating.

Consider the standard 8-hour warehouse shift. Human workers require breaks, shift handovers, and display varying package-per-minute rates depending on fatigue. The F.03 demonstrated a flat latency curve. The speed of sorting at hour 2 was identical to the speed of sorting at hour 29. This is the difference between a biological system and a deterministic loop. When you benchmark labor costs against a flat 30-hour output, the unit economics flip. You are no longer calculating hourly wages. You are calculating the cost of electricity per kilowatt-hour against the depreciation schedule of the hardware. The hardware amortization curve drops off a cliff when the utilization rate hits 100 percent across a 24-hour cycle.

There is also the data generation aspect. 30 hours of continuous, successful operation across three robots yields 90 hours of high-fidelity, real-world telemetric and visual data. This is an MLOps goldmine. Every successful grasp, every minor slip that was auto-corrected, feeds back into the training pipeline. The flywheel effect here is exponential. They are not just sorting packages. They are mining edge-case data at scale. The physical world is the ultimate test set, and Figure is harvesting it faster than anyone else right now.

If you are setting up the ML infrastructure for a warehouse deployment today, you need to rethink your telemetry ingestion. 90 hours of continuous operation generates terabytes of multimodal logs. Video feeds, joint torques, battery thermals, inference latencies. If you do not have a robust data pipeline to filter the noise and only store the edge cases where the confidence score dropped below a threshold, your cloud storage costs will eclipse your labor savings. You need a localized vector database just to handle the short-term memory of the factory floor state.

The F.03 is essentially a walking edge-compute node. When the battery starts to dip, the power management system likely down-clocks the inference chips, reducing the frame rate of the vision models slightly to conserve energy for the actuators. We need to see the latency graphs on the token generation during the final hour of that 30-hour run. Did the sorting speed decrease. Did the confidence threshold widen. The livestream looked steady, which points to an extremely flat power discharge curve and highly deterministic resource allocation.

I benchmark models so you do not blow your budget. The benchmark here shows that the F.03 can sustain continuous industrial operation longer than any standard context window can stay relevant without clearing. It changes the infrastructure requirements for any company planning to deploy embodied agents. The livestream proved the hardware is ready. Tested on prod. What infrastructure fails first when the robots literally do not stop moving.

TLDR: Yes, they're completely different.

A sandbox runs an agent and returns what happened. A QA execution layer runs an agent and returns whether what happened was good enough. Those are not the same question and the output is not the same data.

Outcome analysis without a quality layer is just a log file with better formatting.

The polarity is a sandboxed QA environment for agents, meaning it combines execution sandboxing with quality assessment in a single layer rather than treating them as separate tools, which is the distinction that makes the output actionable for catching regression rather than just confirming task completion.

Hey guys, i was curios, what is the usual setup when having deploy code pattern for model training, so idea is that data scientist run model experiments, different featurization, and just iterate fast on the data on development workspace/environment. Each developer gets its own schema for isolation.

Then when they got something which they want to be promoted, what happens? Of course output of this stage is the training pipeline code, but for example, they did the full hyper-parameter tuning experimentation, so with actual training pipeline code which goes through code quality checks, unit testing, type hinting, do we promote:

a) same hyper-parameters tuning search space (what about cost, variance of possible options etc..)

b) narrowed down search space for tuning

c) parameters of best fitted model

Also do we write this into yaml files within the repo, or there is some better practices where u just fetch ml experiment metadata, or write to UC Volumes, generally interested to see what people are using for this.

Thanks

Hi everyone,

When I started learning Machine Learning, I found plenty of tutorials and courses, but I struggled to find a structured way to practice what I was learning.

So I built **ML Playground**: a hands-on platform designed to help learners progress from fundamentals to advanced topics by writing real code.

**What’s included**
17 structured chapters

140+ interactive coding stations

120+ coding problems with automated test cases

Daily challenges

XP and leaderboard system

The goal is to make ML learning more structured and practice-oriented.

It’s free to start:
[https://mlplayground.in\](https://mlplayground.in/)

I’d love to hear your feedback on:
The learning experience

The curriculum structure

Features you’d like to see added

Thanks for checking it out.

I’m currently working as an ML Engineer, and lately I’ve been thinking about shifting more toward MLOps

My assumption is that companies will still need devops who can deploy / maintain LLLM models bought from other companies

I understand nobody really knows where the industry will end up. I would like to hear from you all to understand what skills are worth investing time into during this uncertain phase instead of just doing nothing?

A thing that surprised me while digging into agent reliability is that a model with 95% accuracy per step sounds excellent. But if your agent takes 10 steps to complete a task, the overall success rate drops to ~60%. And at 100 steps, it’s basically unusable (~0.6%). The failure compounds fast.

Then I came across a few numbers that made this feel less theoretical. Datadog tracked 8.4M AI model request failures in March 2026 and reported that ~5% of AI requests fail in production. A large chunk of these aren’t infra outages, but logic/quality failures that teams can’t properly debug. Similarly, McKinsey in its report said that while many enterprises are experimenting with agents, very few are actually scaling them successfully in production.

The more I look at this, the more it feels like an experimentation infrastructure problem, not a model capability problem. Most teams still test agents in playgrounds/staging and then hope production behaves similarly. But prompts, tools, memory, routing, temperature, context length, fallback logic, etc. all interact in weird ways under real traffic.

Web teams solved this years ago with A/B testing and controlled rollouts. Feels like agent teams need the same thing. Like experiment on live traffic, compare prompt/config variants, isolate regressions, and measure task success over time.

Curious if you agree to this or think there are better ways to solve these production issues.

I started learning Python, and I'm curious, do you have any tips to learn it and how to do it right?

I'm relatively new to MLOps and I've been tasked with productionising feature engineering code (mostly written in SQL) into Lakeflow Spark Declarative Pipelines (SDP) on Databricks.

The current workflow is a bit tedious; DS decides the model is ready, hands me the feature logic (which are huge, complex SQL code with many joins and aggregations for every feature they've ever researched), and based on the features that model actually needs, I slim down the SQL code to only output those features. This is necessary as the project requires features to be served within 1 hour of raw data being ingested, and creating a "master" pipeline for all features that runs continuously to meet the time frame was extremely expensive.

As you can guess, with this workflow, when DS updates their model or adds a feature, I have to manually edit the pipeline code. Sometimes it's a lot of work even for one added feature as there may be a lot of intermediate operations and/or CTEs involved in its computation. I would trace back the original complex logic, which is a PITA.

I'm still new to this, so I would like to hear from this community any advice or solution you may have on approaching this problem, preferably one that integrates smoothly with Databricks.

ChatGPT talked about implementing a framework where DS adds feature metadata to a feature registry, each model gets a config file listing its features, and a parser reads it and auto-generates the pipeline by piecing the feature engineering operations together.

Sounds great, except I still can't seem to wrap my head around the idea of a parser that can reliably assemble the SQL code without including too many unneeded features (as features may be computed together), especially since the code I have is very complex and I still have to reduce joins and nesting in each file such that the pipeline materialized views can incrementally refresh.

We spend an unreasonable amount of time on this sub arguing over whether Qwen-max is beating Llama-3.5 on math evals. It is the wrong metric. I benchmark models so you do not blow your cloud budget, and looking at the current deployment data, the open-weight leaderboard is a distraction. The real split between the US and China is not happening on Hugging Face. It is happening in enterprise procurement.

The US is winning the AI race where it actually matters: commercialization. Here is the data.

Last week, OpenAI quietly dropped a massive signal by launching a $4B deployment venture. Not a research lab. A dedicated deployment company. Their revenue chief stated enterprise adoption is hitting a tipping point. Translation: the raw models are good enough right now, and the new bottleneck is hand-holding legacy businesses through API integrations, compliance routing, and VPC setups. You do not allocate $4 billion just to train a slightly better base model. You spend it to build the infrastructure that forces your models into the operational workflows of Fortune 500s.

When you look at the token economics of enterprise deployment, the strategy is obvious. Caching context for a 100k token prompt across thousands of concurrent corporate users destroys margins if your infrastructure is not custom-built for it. The new deployment push targets dedicated throughput, guaranteed uptime SLAs, and custom hardware setups that standard API tiering cannot handle. This is the unsexy part of AI. It is also the part that prints actual recurring revenue.

Contrast this with the telemetry coming out of China. Look at Alibaba. $BABA has been facing a structural sell-off driven heavily by their massive AI capex paired with a slower monetization narrative in their core market. Technically, they are building the most complete vertically integrated stack outside the US. They have proprietary T-Head silicon feeding into their cloud infrastructure, powering the Qwen models, which directly feed a MaaS platform. It is a highly efficient loop on paper.

But the software monetization is stalling compared to the US enterprise land grab. The Chinese strategy right now leans heavily toward immediate industrial deployment. They are pushing AI into physical workforces and factory floors, with millions of industrial robots already active. The US strategy is pure white-collar enterprise software dominance.

Let us look at the US spending curve. Projected US AI capex for 2025 is floating around $400 billion. The vast majority of that is going toward frontier models and the raw data center grid power required to sustain them. That level of capital expenditure requires an immediate, aggressive commercialization pipeline to justify the burn rate. And the pipeline is executing.

The federal government has quietly become one of the largest AI buyers globally. Government deals do not move like standard SaaS subscriptions. We are talking fixed budgets, rigid procurement cycles, and locked-in vendor relationships. Once a deployment company wires a federal agency or a major healthcare network into a specific ecosystem, the switching costs become permanent.

As an MLOps engineer, when I benchmark latency and token costs across these providers, the actual API inference cost is becoming a rounding error. You can run open-weight models for fractions of a cent per million tokens. But standing up the internal platform to serve it reliably to 10,000 corporate employees securely costs millions. The model layer is commoditizing. The deployment layer is where the moat is being dug.

If you are building right now, stop over-optimizing for a minor bump on an evaluation dataset. Focus on how fast your application can securely parse a messy enterprise data lake. The US is winning because they are treating AI as a standard operating lever, not a research project.

Numbers do not lie. Tested on prod always beats a theoretical benchmark. What is the primary deployment bottleneck in your own infrastructure right now. Is it compliance, inference latency, or raw compute costs.