r/mlops

Hi r/mlops,

I’m a student working on machine-learning experiments, and I kept running into a very ordinary problem: after several runs, I could no longer answer exactly which Git commit, configuration, and Python environment had produced a particular result.

I built RunTrace to address that problem.

RunTrace is a small, open-source, local-first Python CLI that records the reproducibility context around an experiment. Its scope is intentionally narrow: it is not trying to replace MLflow, Weights & Biases, or another full experiment-tracking platform.

It currently records:

• Git commit, branch, detached-HEAD state, and dirty state
• Python, operating system, architecture, and installed package versions
• Optional NVIDIA GPU, driver, and CUDA information
• A YAML configuration file, its SHA-256 hash, and its parsed values
• The command associated with the experiment

A typical workflow looks like this:

pip install ml-runtrace

ml-runtrace init

ml-runtrace snapshot \
  --name baseline \
  --config config.yaml \
  --command "python train.py --config config.yaml"

ml-runtrace list
ml-runtrace show <run-id>
ml-runtrace diff <run-a> <run-b>

Snapshots are stored locally as readable YAML files under .runtrace/runs/. There is no account, server, or automatic upload.

There are also some deliberate limitations:

• It does not execute the recorded command.
• It does not currently track metrics, checkpoints, or model artifacts.
• It records that a Git working tree is dirty, but it does not save source patches.
• Explicit configuration values are stored in the snapshot, so users should inspect a snapshot before sharing it.

The project is still early, and I am trying not to add features without understanding whether they solve a real problem.

I would particularly appreciate feedback on these questions:

1. Does this solve a useful gap, or is it too narrow compared with existing workflows?
2. Is readable local YAML a sensible storage default?
3. What missing metadata or edge cases would prevent you from using it?
4. Is the init → snapshot → list/show → diff workflow understandable?

GitHub:

https://github.com/Corvus-226/RunTrace

Development note: I used Codex as a coding assistant during implementation. I handled the project scope, reviewed the changes, and managed the issue, pull-request, testing, CI, and release decisions. I am mentioning this because I would rather be transparent about how the project was built.

Critical feedback is genuinely welcome. If the idea is redundant, the defaults are wrong, or part of the workflow is unnecessarily complicated, I would rather learn that now than keep expanding it in the wrong direction.

Hey Everyone,

Currently working on a project where I built a system to determine if a clothing item is machine washable or not. Containerized and deployed to AWS to automate scraping and model retraining. Trying to finish up the backend and frontend, and I'm thinking of integrating Grafana and Prometheus. Might post on LinkedIn after (I'm currently job hunting), but posting on linkedin is so embarrassing to me lmao. Let me know what you think. Feedback would be appreciated. Thanks!

Github Repo: https://github.com/sogofunmi/Dryclean-or-No-Dryclean

Conclusion: Thanks for the comments, everyone has been so helpful and generous with the suggestions. I realized that I did not asked my question properly and my main problem went unseen... (Well I'm at fault by opening the conversation by Is there a "TypeScript for Python"?).
I will create a new post with the right problem statement, but let me thank u/ProsodySpeaks, and u/JamzTyson which gave me idea on what to do next.

Cheers!

Hi, sorry for the basic question. I'm coming from strongly/statically typed languages (Kotlin, Go, Rust, etc.), and I was aware that Python is dynamically typed, but given how popular Python is, I expected its typing utilities (type hints + static type checkers) to provide something closer to TypeScript.

I'm working on an ml framework where the main interface has to be Python, and I ran into a magnitude of problem I was not expecting...

Requirements:

• Type checking before a pipeline runs (no values exists yet, just type hints/annotations)
• Type checking during the pipeline run (value and type hints/annotation should match)
• The type hints are used to decide if pipeline components are compatible (similar to LangChain or similar frameworks)
• Require as little setup as possible, so even junior engineers can use the framework safely.

I spent some time trying to implement this using Python's existing typing mechanisms. A few thousand lines of code later, I ended up with a type checker for my specific pipeline system:

https://github.com/trained-by-humans/ml-pipes/blob/main/packages/core/src/ml_pipes/validation.py

And my own type checking utilities:

https://github.com/trained-by-humans/ml-pipes/blob/main/packages/core/src/ml_pipes/_typing/annotation.py

But now I'm wondering:

Am I going way too far here? Is there a much more idiomatic Python approach that I'm completely missing?

And just to be clear: this is only the pre-run check so far. Runtime type checking doesn't exist yet.

Update1: Thanks for highlighting Pydantic, I've considered to use it for runtime since it covers enforcing type hint/annotation on values.

Update2: The TS or other "typing languages", would essentially help with highlighting the compatibility, they answers questions like is input of type A is assignable to parameter of type B, which is very very important for pipeline validation before running the pipeline.

Update3: The type of type checking I need is this, imagine a pipeline like this:

Pipeline([
    Resize((640, 640)),
    Store("resize_transform", source=1),
    Pick(0),
    Normalize(),
    Infer(model_path),
    Extract("output0", as_="preds"),
    Squeeze("preds"),
    Transpose("preds"),
    Slice("preds", slice(None, 4), as_="boxes"),
    Slice("preds", slice(4, None), as_="scores"),
    ArgMax("scores", as_="classes"),
    GatherRows("scores", "classes"),
    ConvertBoxFormat(from_="cxcywh"),
    NMS(conf_threshold=conf_threshold),
    Recall("resize_transform"),
    ProjectBoxes(),
    ToDetections(),
])

I need to make sure the upstream operator output is compatible with downstream input. You can find out more about it in the page (Validation.md under the operator compatibility section)

Please save me from implementing another few thousand lines of code. 😭

I went to bed with a fine-tune running. Woke up eight hours later to a loss that had been NaN since step 300 — the whole night of GPU time gone, and nothing had told me. If you fine-tune, you've lived some version of this: the run that trained at learning-rate-zero the entire time, or quietly ate a dataset with 40 broken rows, and you only find out at the end.

So I built a deterministic linter for training runs — trainproof (pip install trainproof, MIT) — and then I spent days trying to prove it wrong.

One Qwen2.5-3B QLoRA, run five times: once clean, four times with exactly one thing sabotaged. Four it caught instantly. One fooled it completely — and that one taught me the most.

It wasn't the NaN run. It wasn't the fp16 overflow. It wasn't the learning rate cranked 100× too high (that spiked the gradient norm to 2650× the median — caught in seconds).

It was shuffled labels. Pure garbage — a dataset that literally cannot be learned.

That run reduced its loss by 62%. On the curve it looked like textbook-healthy training. It was learning absolutely nothing — just memorizing the statistics of noise, which any network will happily do. From its own loss curve it is indistinguishable from a real run. No single-run, loss-only rule can catch it. So instead of pretending my tool is magic, I wrote that limitation straight into the README — and added a compare mode, because the failure is visible the moment you put it next to a known-good run.

That's the whole philosophy: no ML judging ML, no "87%-confidence" scores. Every check is a deterministic rule that either fires or doesn't, and every finding cites the exact numbers behind it. When it can't be sure, it says so instead of guessing.

Across a run's life:

• Before a single GPU-second — lints the dataset + tokenizer (malformed JSONL with the line number, empty rows, duplicates, missing eos_token, pad==eos, over-length samples). Exits non-zero → straight into CI.
• During training — one-line HuggingFace callback. Warns by default; flip on stop_on_fail and it aborts a doomed run itself. In testing it killed a diverging run at step 20 of 300 — 93% of the scheduled steps never ran.
• After — reads the log: diverged / flatlined / NaN'd / spiked.
• vs a baseline — the ratio rules that catch the shuffled-labels case.

All five sabotaged runs' real logs ship in the repo (examples/gallery/) with a 15-run / 3-seed evidence matrix — reproduce every verdict yourself. Reads plain logs (HF trainer_state.json, Coqui, JSONL/CSV), doesn't import torch.

Repo: https://github.com/Mormolykos/trainproof · pip install trainproof

Honest question: what's burned your GPU hours? If a deterministic check would've saved you, tell me — it goes in, with credit.

Agent context retrieval is often treated as model behavior. In production, we found that it behaves more like a data-plane architecture decision.

We compared two approaches:

• Retrieve Slack, Notion and Linear data through official MCP integrations during each agent run.
• Pre-sync the permitted data and mount it into the agent sandbox as files.

The mounted implementation was Locality Cloud, which I work on.

The evaluation used 20 cross-application scenarios with three paired trials each. We ran six AWS t3.large instances, kept the agent harness, model, prompts and machines consistent, and performed 180 blind comparisons of the outputs.

Compared with runtime MCP retrieval, the mounted setup:

• Produced the preferred answer in 70% of scenarios.
• Reduced LLM costs by 27%.
• Reduced end-to-end latency by 32%.
• Required 61% fewer tool calls.
• Used roughly 40% fewer tokens.

The traces suggest that the agents weren’t reasoning substantially faster. They were spending less time traversing application data.

In one scenario, the agent had to reconcile product launch risks across Slack, Linear, Notion and a Git repository. One evidence-gathering stage took roughly 0.3 seconds using parallel filesystem operations. The MCP setup spent about a minute on the same stage, making 21 calls with approximately 30 seconds of tool-call time.

The broader MLOps lesson for us is that mounted file system context isn’t simply a cache. It becomes a production data plane with its own requirements:

• Freshness: changes need to arrive through webhooks, polling or a pre-run synchronization boundary. Staleness must be observable.
• Permissions: each sandbox should receive only the sources and subtrees required for that run, without broad application credentials.
• State: remote state, mounted state and the last synchronized state must be tracked separately so pulls, writes and conflicts are unambiguous.
• Write review: agent edits should produce an inspectable operation plan before they are synchronized back to the source.
• Recovery: interrupted writes need journaling, idempotency and explicit conflict handling rather than silent retries.

This architecture also creates new operational costs: connector maintenance, synchronization lag, storage, conflict resolution and recovery testing.

We still expect live APIs or MCP to be preferable for transactional actions, narrow lookups and data that cannot tolerate synchronization delay. The emerging pattern looks less like “files instead of MCP” and more like two planes:

• A mounted context plane for broad, read-heavy discovery and synthesis.
• A live action plane for transactional operations.

Locality Cloud is our managed implementation of the mounted context plane, with an on-premises option for organizations that need to keep the synchronization layer inside their environment.

Full details with analysis, traces and scenario-level results:

https://www.locality.dev/blog/locality-why-filesystems-perform-better-than-mcps-for-production-agents

How are teams operating production agents separating their context plane from their action plane? If you materialize application data before execution, how do you handle freshness, permissions and failed synchronization?

I've been working on an open-source project called UnFlow:

https://github.com/UnFlow-Labs/mlunflow

The idea is pretty simple:

Most ML experiment tracking looks like a list of independent runs usually stored in a table:

run_001
run_002
run_003
run_004
...

But in practice, experiments are usually related.

You change the learning rate, then the number of epochs, then the model, then some preprocessing code. Eventually you have hundreds of runs, but it's surprisingly difficult to answer:

• What actually changed between these two experiments?
• Which experiments are essentially the same computation?
• Have I already run this experiment before?
• How did I get from experiment A to experiment B?
• Can I navigate the history of my experiments rather than just search through runs?

Unflow simply detect code changes in a Python function (limitation that for it is just a single function) and arguments that are passed to this function to build a graph where nodes are "states" and edges are transformations "what has changed", a new state is not added to the graph or executed expect if it has a transformation.

The project is still early, so I'm much more interested in feedback than pretending this is a finished product.

I'm particularly curious about three things:

1. Does the "experiments as a graph" abstraction make sense to you?
2. Do you currently run into problems with duplicated/redundant experiments?
3. If you could see the complete lineage of your ML experiments, what would you want to query or visualize?

Repo: https://github.com/UnFlow-Labs/mlunflow

I'd love to hear how other people currently manage experiment lineage and whether this solves a real problem for you.

My original post got removed from another sub, so reposting here since I still wanna know what people think

Read a TechCrunch article recently talking about a $400M loan General Compute received using specialized SambaNova inference chips as collateral instead of GPUs. This surprised me because I'd always assumed GPUs were the obvious choice for this kind of financing.

There seems to be a shift from training-heavy infrastructure to inference-first workloads. This financing announcement got me thinking about whether investors are starting to put more weight on cost-efficient infra to run open-source AI models instead of just funding expensive frontier models from the big names. Investors are willing to back alternative hardware providers, which could put more pressure on Nvidia's dominance. Open-source models are clearly getting stronger. I'm curious whether this is the start of a bigger shift in how AI infrastructure gets financed and deployed.

Aquifer is an open-source local control plane for AI workloads and MCP infrastructure. It provides durable queues, bounded concurrency, fairness controls, and dynamic pacing for bursty traffic patterns common in agent systems.

It also experiments with the Aqueduct Protocol, a stream and webhook-based coordination protocol that dynamically communicates flow state through headers, allowing clients to scale traffic up or down at a controlled pace instead of relying solely on static rate limits. The project also includes an encryption and identity protocol that uses public-key verification, reducing the need to store shared secrets in a database. The goal is to make agent and MCP traffic more resilient to overload, retries, and traffic spikes.

Repo: https://github.com/rjpruitt16/aquifer

I have been working on TraceML, a local-first runtime diagnostics tool for PyTorch training.

The latest work is focused on distributed runs: making multi-rank / multi-node training easier to inspect after the run finishes. The idea is to produce a compact performance summary for each run, including:

- step time breakdown
- dataloader overhead
- compute vs wait time
- GPU memory behaviour
- rank skew / stragglers

The goal is more of a first-pass regression check: did this run get slower, and where?

For people running DDP/FSDP jobs: what distributed performance issues do you usually miss until too late?

If you have run into these kinds of issues, I would love feedback on what signals would make a distributed training summary actually useful.

Tool info: https://github.com/traceopt-ai/traceml

I’ve been working on MARGINAL, an open-source governance layer for coding agents. If you use Codex, I’d really appreciate people trying it on real work and telling me where it helps, where it gets in the way, or where the design is wrong.

I’m especially interested in: technical criticism, bad cases, and reproducible failures.

• GitHub: https://github.com/SignalLayerLabs/Marginal
• GitHub Site: https://signallayerlabs.github.io/Marginal/
• Live Demo: https://signallayerlabs.github.io/Marginal/demo/#demo

The idea is simple: agents are good at taking actions, but not always good at deciding whether the next action is still worth the compute.

MARGINAL watches the trajectory and looks for things like repeated actions, weak progress, redundant verification, and low-value continuation. It can run in Shadow Mode first, so it observes and records what it would have done without blocking anything.

Current focus is reliability, not just token reduction.

A few core pieces:

• local-first trajectory and evidence tracking
• deterministic reason codes and hashes for decisions
• governance overhead measurement
• replay and benchmark support
• Shadow Mode before enforcement
• Earned Enforcement: MARGINAL has to prove it is reliable on a repo before it gets permission to block or redirect the agent
• automatic fallback to Shadow Mode if confidence degrades

I’m also working on the next layer now: counterfactual evaluation and intervention regret.

The goal is to answer a harder question than “did MARGINAL stop something?”:

Would the agent actually have done better if MARGINAL had stayed out of the way?

That’s the part I think matters if this is going to be useful beyond being another loop detector or token limiter.

Letting an agent optimize a training run without me present usually beats what I would have done by hand, and often it would find more interesting (unfortunately) solutions than I would have tried.

So the question that I am trying to understand now -- what it means to properly write loops (or now graphs lol), and where my value actually is.

I started converging to the workflow where I would spend 1-2 hours carefully designing the optimization objective, goals and constraints, and then just let my Claude Code grind on it for days. I found it very important to have a clear separation of the evaluation code and the optimizable code. So that if I trust the evaluation and I know the agent can not change - I will trust the result the agent produced, so hallucinations is not a problem anymore.

I tried to formalize this philosophy in a skill + CLI library, where Claude helps me build a bulletprrof evaluation environment first (I call it a hill), and then the agent would "climb" it. Hills have a few mechanisms to make sure the agent can not just modify the evals mid-run. You can try it here: https://github.com/autolab-ai/hills (critical feedback is very welcome!).

Curious what everyone's thoughts are, where you see your place in todays workflows, how you design them etc? Do you think this foced separation of the evals and optimiable code is valuable?