u/GlitteringAngle8601

I'm trying to figure out real observability for multi-agent systems, not just single models. Flat logs don't cut it once agents are calling tools, spawning sub-agents, and hitting real systems with side effects.

I'm tracking agent decisions (what it was given, what it picked), tool and API calls (params, latency, errors), and end-to-end traces across agents. There's also a shared session where multiple agents and humans collaborate, so I tag spans with both a trace ID and a session ID.

Metrics like latency and call volume are free but don't tell you why an agent made a decision or which step caused a failure. That needs parent-child span structure plus some captured reasoning. Reasoning capture is messier than it sounds though. With reasoning models you usually get a summarized trace, not the real chain of thought, and what's exposed varies by provider.

Outcome labeling is the part people skip. Volume and latency show up automatically. "Did this trace actually succeed" doesn't, someone has to apply that label, whether it's rules, human review, or LLM-as-judge. Judges have their own issues: inconsistent across runs, costly at scale, and prone to missing the same failure class the agent itself missed.

Biggest open question for me is where instrumentation should live. Network-level interception is framework-agnostic but you lose semantics (was that a plan step or a tool call?). SDK-level gets you semantics but means per-framework work and breaks across mixed runtimes.

Anyone running this across multiple run times in prod. How are you splitting the instrumentation layer, and what's your sampling approach once trace volume gets expensive?

It's impossible to put down..

Run multi agent systems both ways and have strong opinions at this point.

Shared memory simplifies individual agents no large contexts passed through handoffs. In practice it introduces race conditions, stale reads, state bleed between runs, and coupling that's invisible at the code level until something breaks in production.

Explicit handoffs are more verbose but every interaction is auditable. You can see exactly what moved between agents, when, and in what format. That traceability has been worth more than the verbosity cost every single time something went wrong.

Current default: explicit handoffs, shared memory only for truly global state that changes infrequently, and any shared mutable state treated as infrastructure with its own reliability requirements.

Has anyone actually made shared memory work reliably at scale or is everyone quietly moving away from it?

when i finally got to the front, the cashier lady said " sorry mate, we are out of sausage rolls." I said, "That is fine, I shall just wait." She looked at me confused " but we are out. there is nothing to wait for." I said "NO, I mean I shall wait here. I have already invested 20 minutes. I am not leaving empty handed." She called security