One failure mode I do not see discussed enough in LLM observability is absence. If a trace contains no policy check, did the agent bypass it, did the instrumentation drop it, was it sampled out, or is the viewer not authorized to see it? Those are different operational states, but many dashboards render them as the same blank space.

We handle that boundary by treating observability as a contract rather than a bag of spans. For a completion claim, the contract names the evidence that must exist: the requested intent, contract and source versions, tool calls, validator results, and resulting authoritative state. A required surface can be present, explicitly not applicable, unavailable, or missing. Missing does not become a pass. The worker's trace is useful for diagnosis, but a separate verifier decides acceptance from the retained artifacts.

The same principle applies to the observability system itself. A sentinel flow should produce a known span or receipt. If that artifact disappears, the detector failed before anyone interprets silence in production. How are people modeling negative evidence? Can your system distinguish "the action did not happen" from "we cannot prove whether it happened"?

I keep seeing agent eval reports that pin the model version, temperature, prompt, and dataset while treating tool definitions as incidental plumbing.

That boundary does not hold for an agent. Rename a tool, change a parameter description, add a default, widen an enum, or alter the return shape, and the model is now reasoning over a different interface. The backend may remain API-compatible while tool selection and result interpretation change. A passing run from the old harness is not automatically evidence about the new one.

I think the reproducible unit needs to include the model, prompt, context bundle, tool names, parameter schemas and defaults, permissions, return schemas, and validator versions. Semantic changes should mint a new evaluation bundle and rerun the fixture. Cosmetic edits can remain compatible only when they cannot affect resolution.

Does anyone here version tool schemas alongside eval fixtures? If not, how do you tell a model regression from a harness change?

After a few hours, a coding-agent chat contains everything except a reliable answer to what is actually true. It has the original request, three possible approaches, a correction, a half-finished branch, and a confident summary written before the tests ran.

This is manageable with one small task. It falls apart when several agents work across multiple sessions. We learned to keep the plan outside the conversation. The durable plan says what is doing, done, blocked, and next. At handoff, we reconcile those claims against the actual pull requests, checks, and system state. If the chat says finished and the evidence says otherwise, the plan gets corrected.

The chat still matters. It is where exploration happens. It just does not own status or intent.

I think a lot of “agent memory” problems are actually authority problems. We keep trying to make the model remember more when the system needs one governed place to record what was decided and what is true now.

For people running agents across sessions, what survives the chat? A plan, an issue tracker, an event log, or mostly a summary generated at the end?

Most agent-memory systems store conversations, summaries, embeddings, or workflow checkpoints. Those are useful, but they usually lose the exact relationship between the goal, selected context, repository state, attempted actions, failures, validation, and accepted outcome.

I am exploring a different memory object for software agents: an immutable, typed causal history joined to versioned artifacts.

The rough model is:

goal
  -> selected context
  -> exact files and symbol versions observed
  -> tools and delegations
  -> attempted changes
  -> failing validation
  -> corrective action
  -> accepted outcome
  -> resulting artifact versions

Small facts, identities, correlation, declared causation, and lifecycle transitions would live in an event ledger. Larger objects such as source files, diffs, context bundles, and model outputs would be content-addressed artifacts referenced by those events.

Graph, vector, relational, search, and hot-cache databases would remain useful, but only as rebuildable projections. The ledger says what happened. Artifacts preserve what existed. Projections make that history useful for a particular query.

The part I find most interesting is treating a successful historical subgraph as an exemplar. Instead of retrieving text that resembles the current task, the system could recover a previously successful transformation, compare its original dependency graph with the current repository, identify changed assumptions, and reuse only the structure that still passes current validation.

This is a position paper and research direction, not a claim that the full architecture is implemented or that its individual ingredients are novel. Event sourcing, content-addressed storage, code graphs, vector retrieval, tracing, and workflow replay already exist. The question is whether joining them under one authority model creates a meaningfully better memory substrate.

The smallest prototype I can see is one real coding-agent task: capture durable execution events and exact source state, join them into an execution and code-evolution graph, declare one successful slice as an exemplar, then attempt to adapt it after the repository changes.

I would especially value pushback on three questions:

1. Is a causal subgraph actually a useful first-class memory object, or can existing episodic-memory representations provide the same value?
2. What is the minimum event set needed to reconstruct a software-agent transformation without creating an unusably expensive trace system?
3. Where should the boundary sit between immutable historical facts and mutable semantic interpretation?

The most embarrassing useful failure in our agent workflow returned success almost immediately.

An orchestration command exited cleanly, printed success, and went green. It had also executed zero phases and published no useful work. We had accidentally defined completion as "the process did not crash." That sounds ridiculous written down, but it was easy to miss because every visible signal said the run was healthy.

That failure is why I no longer treat a success message as evidence that an agent completed the work. I want evidence at the workflow boundary: which phases ran, which artifacts were produced, what state changed, and whether somebody other than the process making the claim can verify it. The exact evidence depends on the task. It might be passing tests, a deployed service responding correctly, or a human accepting a draft. It just cannot be the agent saying it is done.

What is the best false-positive success you have seen from an agent or automation workflow? The one where everything was green and nothing useful actually happened.

We recently analyzed 1,521 workflow records collected across three weeks. After removing machine payloads and empty material, the useful corpus contained 780 human prompts across 86 sessions.

The most valuable unit was not the prompt. It was the correction episode: what the task asked for, what the agent did, what the operator corrected, whether the same correction appeared elsewhere, and what evidence existed about the outcome.

Repeated corrections exposed several stable patterns. Plans were expected to survive the conversation. Completion reports were treated as claims that needed independent evidence. Work was divided along boundaries that could be accepted separately. Cleanup and handoff were part of delivery rather than optional follow-up.

That does not prove those practices improve outcomes. This was one operator, one tool environment, and a bounded period. Some assistant turns were unavailable, and transcripts cannot prove that reported work actually succeeded. The defensible result is narrower: operational conversations can reveal recurring rules and missing workflow controls that a retrospective interview tends to smooth away.

I think an agent evaluation dataset should include human corrections, retries, overrides, and false-completion reports, not only final answers. Those traces show where the surrounding system still relies on judgment it has not made explicit.

Are any of you mining correction patterns across sessions, or are your evaluation datasets still mostly task-and-answer pairs?

One of the most expensive mistakes I made with coding agents was treating the completion message as evidence.

Claude Code would finish a task and tell me the tests passed, the bug was fixed, and the requested files were updated. The summary was confident and specific. It was also sometimes wrong.

That does not require dishonesty or a uniquely bad model. A coding agent is good at producing the expected shape of a completion report. If it ran the wrong test, misunderstood the output, skipped a requirement, or simply lost track of an earlier failure, the final summary can still sound exactly like a successful one.

The useful distinction is simple:

• “The tests pass” is a claim.
• The test command, exit code, and output are evidence.
• “The feature works” is a claim.
• Exercising the behavior against the acceptance criteria is evidence.
• “I only changed the requested scope” is a claim.
• Reading the actual diff is evidence.

Once I started treating every completion report as a set of claims to verify, coding-agent work became much easier to reason about.

Trust is not a prompt setting

The first instinct is to improve the prompt: tell the agent to be careful, require a checklist, or say that it must not claim success without running tests.

Those instructions can help, but they do not make the report independent of the work. The same actor is still producing the change, deciding whether the change is correct, and describing the result. It is grading its own homework inside the same context that produced the answer.

The fix is a separate verification path.

That can be CI, a deterministic command run after the worker exits, a second agent reviewing the actual diff and artifacts, or a human checking the result. The important property is not that the verifier has a different name. It is that the verification starts from the produced artifacts and the original acceptance criteria, rather than inheriting the producer's summary as truth.

A second agent that receives only “the first agent says everything passed” is not independent. A second agent that receives the task, the diff, and the command output has a chance to be.

What I verify before accepting a coding-agent change

My minimum bar is now:

1. Inspect the diff and confirm it stayed inside the requested scope.
2. Run the relevant test, lint, or type-check command from the resulting state.
3. Check that the command actually covers the behavior that changed.
4. Compare the result with the original acceptance criteria, not the agent's rewritten version of them.
5. Record anything skipped, including why it was skipped.

The third item matters more than it looks. A green test is weak evidence if it never reaches the changed behavior. Agents can add a test that passes without proving the requirement, run a nearby test instead of the relevant one, or preserve an existing test suite while changing behavior the suite never covered.

Verification therefore cannot stop at “a command exited zero.” The command has to be connected to the claim.

Verification is expensive, but skipping it is not free

Reading diffs and rerunning checks takes time. When several workers finish together, verification becomes a real share of the operator's workload.

The alternative is not zero cost. It is deferred cost.

A mostly-correct change gets accepted. The next task builds on it. Two more tasks inherit its assumption. The original mistake finally appears several layers later, when it is much harder to locate and more expensive to unwind.

That changed how I think about agent throughput. The useful number is not how many tasks an agent can report as finished. It is how many independently verified changes I can accept without creating rework downstream.

For people running Claude Code on real repositories: what evidence do you require before you believe a completion report?

I’ve been working on something over the past year that’s turned into a distributed runtime for AI applications, and I’d love feedback from people with more experience in distributed systems than I have.

My background is mostly mobile engineering, so I didn’t come into this with years of distributed systems experience. I approached the problem from first principles, kept iterating, and eventually ended up with an architecture that feels a bit like an operating system for distributed applications.

The core idea is that independent runtimes communicate through versioned contracts and events. Runtimes execute work, reducers own state transitions, and everything is designed to be replayable and deterministic. One design goal was to make the runtime completely independent of any particular model or provider. Models are treated as interchangeable compute resources, whether they’re running locally, self-hosted, or through cloud APIs. As long as a model satisfies the contract, the orchestration layer doesn’t care where it came from.

I’m not claiming I’ve invented something entirely new, and I’m sure there are systems that solve similar problems in different ways. That’s actually why I’m posting.

I’d love to know:

* What existing systems or papers does this remind you of?

* Where do you think this architecture is weak?

* What failure modes or scaling issues would you immediately worry about?

* If you were designing this today, what would you do differently?

I’m happy to share diagrams, architecture docs, or code if people are interested. I’m looking for honest technical feedback from people who’ve built distributed systems before.

I’ve been working on something over the past year that’s turned into a distributed runtime for AI applications, and I’d love feedback from people with more experience in distributed systems than I have.

My background is mostly mobile engineering, so I didn’t come into this with years of distributed systems experience. I approached the problem from first principles, kept iterating, and eventually ended up with an architecture that feels a bit like an operating system for distributed applications.

The core idea is that independent runtimes communicate through versioned contracts and events. Runtimes execute work, reducers own state transitions, and everything is designed to be replayable and deterministic. One design goal was to make the runtime completely independent of any particular model or provider. Models are treated as interchangeable compute resources, whether they’re running locally, self-hosted, or through cloud APIs. As long as a model satisfies the contract, the orchestration layer doesn’t care where it came from.

I’m not claiming I’ve invented something entirely new, and I’m sure there are systems that solve similar problems in different ways. That’s actually why I’m posting.

I’d love to know:

* What existing systems or papers does this remind you of?

* Where do you think this architecture is weak?

* What failure modes or scaling issues would you immediately worry about?

* If you were designing this today, what would you do differently?

I’m happy to share diagrams, architecture docs, or code if people are interested. I’m looking for honest technical feedback from people who’ve built distributed systems before.

I’ve been working on something over the past year that’s turned into a distributed runtime for AI applications, and I’d love feedback from people with more experience in distributed systems than I have.

My background is mostly mobile engineering, so I didn’t come into this with years of distributed systems experience. I approached the problem from first principles, kept iterating, and eventually ended up with an architecture that feels a bit like an operating system for distributed applications.

The core idea is that independent runtimes communicate through versioned contracts and events. Runtimes execute work, reducers own state transitions, and everything is designed to be replayable and deterministic. One design goal was to make the runtime completely independent of any particular model or provider. Models are treated as interchangeable compute resources, whether they’re running locally, self-hosted, or through cloud APIs. As long as a model satisfies the contract, the orchestration layer doesn’t care where it came from.

I’m not claiming I’ve invented something entirely new, and I’m sure there are systems that solve similar problems in different ways. That’s actually why I’m posting.

I’d love to know:

* What existing systems or papers does this remind you of?

* Where do you think this architecture is weak?

* What failure modes or scaling issues would you immediately worry about?

* If you were designing this today, what would you do differently?

I’m happy to share diagrams, architecture docs, or code if people are interested. I’m looking for honest technical feedback from people who’ve built distributed systems before.

We're working on a problem that sits at the intersection of distributed systems and ML deployment: how do you get multiple AI agents — backed by different models, running across different machines — to coordinate work, verify each other's output, and produce results you can actually trust?

OmniNode is our answer. The architecture is:

- Contract-driven task delegation (structured specs, not prompt chains)

- Multi-model routing — local inference (Qwen3-Coder-30B on RTX 5090, DeepSeek-R1 on M2 Ultra) alongside cloud models, routed by task characteristics

- Deterministic execution with event sourcing (Kafka) and replayable state

- Evidence-backed completion — work isn't "done" because an agent said so, it's done when durable verification passes

The core insight we're building on: as agent autonomy increases, reproducibility and enforceable correctness matter more, not less. Prompts don't scale. Contracts do.

Looking for engineers who think in systems, care about how things fail under real conditions, and are comfortable working across the ML/infrastructure boundary.

Fully remote. DM me with something you've built or something interesting you've been experimenting with.

I don't really care where you went to school, what degree you have, or whether you can reverse a binary tree on a whiteboard.

I care whether you:

- Think in systems

- Are deeply curious

- Obsess over correctness and failure modes, not just happy paths

- Can learn fast

- Can use AI tools effectively

- Ship constantly

- Explore problems instead of waiting for instructions

- Understand architecture, tradeoffs, and how things break

- Actually build things outside of work

We're building distributed systems for AI agent orchestration — contract-driven delegation, verification, and autonomous pipelines. The people who thrive here naturally go down rabbit holes, prototype constantly, and care about how systems behave under real conditions — not just whether the code compiles.

If that sounds like you, DM me or drop a link in comments:

- something you've built

- something you're proud of

- or something weird/interesting you've been experimenting with lately

GitHub > résumé. Curiosity > credentials. Systems thinking > LeetCode.

edit: This is a paying gig. Added because u/Organic-Afternoon-50 didn't think this was clear.