r/LLMObservability

If you're building a document-processing pipeline today, does it actually make sense to send every stage through the same high-end multimodal model? My instinct is that a lot of document work doesn't need the most capable model.

For example:

• Clean PDFs / straightforward OCR: traditional OCR, direct text extraction, or a lightweight model may be enough.
• Parsing and simple extraction: a faster, lower-cost model such as Gemini Flash-class models may handle this well.
• Handwriting, poor scans, complex tables, or ambiguous fields: this may be where you route to a more capable multimodal/reasoning model.

The part I'm unsure about is whether the accuracy and cost advantage of model routing is actually worth the orchestration complexity in production.

Here’s how I’m thinking about the trade-offs:

• Accuracy: One model gives you more consistent behavior, but it may be overkill for simple documents and weaker on certain edge cases. Multi-model routing lets you optimize by document type or task, but poor routing decisions can hurt accuracy.
• Latency: One model means fewer routing steps and simpler execution. Multiple models can keep easy documents on faster models, but retries and escalations may add latency.
• Cost: One model is easier to predict, but expensive if a premium model handles everything. Routing can reduce cost significantly if most documents can stay on lightweight models.
• Privacy: One provider/model can simplify governance and data handling. Multiple providers add complexity, although routing could also keep sensitive documents on private or internally hosted models.
• Fallback behavior: With one model, a retry may simply reproduce the same failure. With routing, low-confidence outputs can escalate to another model or eventually to human review.
• Maintenance: One model is much easier to operate. Multi-model pipelines require more evals, routing logic, monitoring, version management, and regression testing.

I'm especially interested in the fallback strategy.

Would you use:

small model → larger model → different provider → human review

or simply:

one strong model → human review when confidence is low?

And what would you use as the routing signal: OCR confidence, image quality, handwriting detection, document type, extraction confidence, schema validation failure, or something else?

For anyone running document AI at meaningful volume: has multi-model routing actually reduced cost and improved accuracy, or does the added complexity outweigh the benefit?

Most LLM observability setups treat a tool call as successful if the transport succeeded. With MCP that's wrong: the protocol returns CallToolResult.isError=true inside an HTTP 200. Transport says fine, the tool actually failed, and your error-rate panel stays flat while the agent quietly retries or hallucinates around the failure.

I've been building opentel-mcp, an OpenTelemetry instrumentation layer for MCP servers, to close that gap. It's at ~1,500 npm downloads now. What it does:

Correct error semantics. isError=true gets marked status: ERROR with error.type: tool_error, so failed tool calls actually appear in error rate.

Two-axis outcome model. One axis is ToolOutcome (did the tool do its job), the other is ObservationIntegrity (can you trust what you observed). These get conflated constantly — a schema-drifted response isn't the same failure as a timeout, and collapsing them makes dashboards lie.

Cost and token attribution across 19 models with per-model overrides for negotiated rates, plus embedding pricing.

W3C trace context propagation, so the server span becomes a child of the agent span instead of an orphaned tree.

One finding worth flagging for anyone on stateless HTTP transport: cross-call detection — anything that counts patterns across calls, like agent thrash detection — cannot work when every request gets a fresh instance. I hit this instrumenting my own agent and filed it as a library gap rather than pretending it worked. It's a structural limit of stateless transport, not a bug you can patch around.

npm: https://www.npmjs.com/package/opentel-mcp

Docs: https://opentel-mcp-site.pages.dev/

Curious what others here are doing for tool-call-level observability, especially if you've solved the stateless correlation problem in a way I haven't thought of.

AI coding agents are getting increasingly autonomous — writing code, modifying files, running tests, interacting with tools, and in some setups even deploying changes.

But we've also seen discussions around:

security vulnerabilities introduced by AI-generated code

confidently incorrect decisions

agents having too much access/authority

developers trusting fluent output more than they should

So I'm curious where people actually stand:

Would you trust an AI coding agent to ship production code without a human reviewing it?

Assume the agent has access to the repo, tests, and normal development tooling.

I'm especially curious about the German/European perspective — engineers, founders, developers, security people, etc.

If you're working in Germany, what would make you trust an AI agent in production?

View Poll

One failure mode I do not see discussed enough in LLM observability is absence. If a trace contains no policy check, did the agent bypass it, did the instrumentation drop it, was it sampled out, or is the viewer not authorized to see it? Those are different operational states, but many dashboards render them as the same blank space.

We handle that boundary by treating observability as a contract rather than a bag of spans. For a completion claim, the contract names the evidence that must exist: the requested intent, contract and source versions, tool calls, validator results, and resulting authoritative state. A required surface can be present, explicitly not applicable, unavailable, or missing. Missing does not become a pass. The worker's trace is useful for diagnosis, but a separate verifier decides acceptance from the retained artifacts.

The same principle applies to the observability system itself. A sentinel flow should produce a known span or receipt. If that artifact disappears, the detector failed before anyone interprets silence in production. How are people modeling negative evidence? Can your system distinguish "the action did not happen" from "we cannot prove whether it happened"?

I’ve been thinking a lot about false confidence in LLM/agent debugging tools.

When two agent runs differ, it’s tempting for an observability tool to pick the first suspicious-looking span and present it as “the cause.”

But in a lot of cases, the structural evidence is ambiguous:

- repeated tool calls can be hard to align safely

- some captured content may be unavailable or redacted

- traces may be incomplete

- later differences do not necessarily imply causality

I’ve been experimenting with a different rule in a local-first run-comparison tool I’m building:

If the evidence is not sufficient to establish an investigation starting point, return `uncertain` instead of selecting the most plausible-looking location.

The tradeoff is obvious: you sometimes give the user less information.

But the benefit is that the tool does not silently convert ambiguity into confidence.

I currently test this with a deterministic regression corpus where ambiguous cases are expected to fail closed rather than guess.

I’m curious how people working on LLM observability think about this:

Would you rather have a debugging tool return a weaker/uncertain result more often, or make a best-effort guess as long as it explains why?

And what kinds of evidence would you personally require before calling something an actual “root cause” rather than just an investigation starting point?

I started building an external memory system because I got tired of re-explaining old projects to an LLM every time context disappeared.
At first I thought the problem was just memory.
It wasn’t.

The bigger problem was that memory, research, live repository state, and execution results are different kinds of truth.

If remembered context says the system works one way, but the live repository changed three commits ago, the repo should win.

If a research paper suggests a better technique, that does not mean the project should silently adopt it.
If something happened in chat, that does not automatically make it durable project knowledge.

That pushed me toward separating the system into different authority classes instead of dumping everything into one retrieval store.
Memory keeps durable project knowledge.
Research holds outside evidence.
Repositories tell me what the implementation actually is now.
Execution records what really happened when something ran.
Chat stays temporary unless something earns promotion.

Retrieval had to change too.
Instead of asking, “What looks semantically related?”, the first question became:

Which exact project or scope owns this request?
Then retrieve the smallest useful packet from the source that actually owns that kind of truth.

That has helped a lot with stale context and cross-project bleed.
The rule I keep coming back to is:
Experience is allowed to change the system, but it has to earn the state transition first.

An experiment can become evidence.
Evidence can become a candidate.
A candidate can become memory, a skill, a workflow, or canon.
But not automatically.
Otherwise you eventually build a system where “the model found something similar” quietly turns into “this is now true.”

I’m curious how others handling long-running agents approach this.
Do you treat memory, live state, research, and execution evidence as separate authority classes, or does it all eventually end up in one retrieval layer?

I was invited to share TraceMotive here earlier, and I just released v0.4.0.

TraceMotive is a local-first OSS tool for comparing AI agent executions and helping answer:

“Where did these two runs first diverge in observed behavior?”

The main change in v0.4.0 is that the comparison result is now organized as an investigation workflow rather than just a trace diff:

• Look here — the first evidence-supported place to investigate
• What changed — conservative structured JSON differences
• Evidence — what was actually observed
• Next — direct navigation to the corresponding left/right spans
• What TraceMotive does not know — explicit limitations and uncertainty

Other additions in v0.4.0:

• Minimal investigation cockpit
• Conservative structured JSON diff
• Direct left/right span navigation
• Additive /api/v4 comparison contract
• First-run onboarding
• Deterministic identified and uncertain demo scenarios
• Fresh-checkout / installed-wheel E2E validation
• PyPI Trusted Publishing

One design constraint I’m intentionally keeping is that TraceMotive does not treat the first observed divergence as proof of root cause.

If traces are incomplete, capture is unavailable/redacted, or repeated spans cannot be safely aligned, the result should remain uncertain rather than force a match.

In the current 30-scenario adversarial corpus:

• 15/15 expected confident behavioral-divergence cases were identified
• 14/14 supported investigation starting points were identified
• 0 false-confident behavioral-divergence results
• 0 false-confident investigation-starting-point results

Those numbers are corpus-scoped, not a universal accuracy claim.

Install:

pip install "tracemotive[server]==0.4.0"

I’m a high-school student building and maintaining this with heavy use of AI coding tools.

I’d especially appreciate feedback from people working on LLM observability about the investigation flow, uncertainty handling, and whether this kind of “first supported divergence” view would actually save time in real debugging.

Midweek thread. Drop whatever you are building, breaking, or staring at.

Most of the bugs that eat our week come from around the model. A retrieved chunk that was quietly wrong. A tool that started returning a new field. A prompt someone tweaked for better results. Every dashboard stays green the whole time.

So what is eating your week right now? Extra curious about the one you were sure was the model, and it turned out to be something upstream.

First time instrumenting an LLM app and not sure what to log? Ask here.

Unpopular around release week, but here it is. Most of the agent behaviour I have had to fix was not the model being dumb. It was the scaffolding: how tools were described, what got put back into context after a failure, how many steps were allowed, what happened on a timeout.

The test I use now is to debug on a weaker model on purpose. If the flow only works on the best available model, what I have is not a working agent, it is a model compensating for my plumbing. When the frontier model is the only thing holding the loop together, the next behaviour change in that model is going to break me and I will have no idea why.

The uncomfortable part is that harness work is boring. Nobody writes a launch post about tool descriptions that do not overlap, or about the retry policy. But that is where the wins were.

I am not claiming model quality does not matter. It obviously does, and there are tasks that simply do not work below a certain capability. I am claiming the ratio is nothing like what the discourse suggests.

Where do you think the line actually sits? Curious if anyone has the opposite experience, where swapping the model fixed something the harness could not.

I start with the decision that would hurt if the model got it wrong, then build pairs around that boundary. Random edge cases haven’t been nearly as useful.

For a support-ticket router with auto_route and needs_review, I keep:

• one obvious positive for each route

• one near-miss from the neighboring route

• one underspecified ticket that should abstain

• one ticket with conflicting cues

• one real failure from production

Then I score auto-route precision and needs_review recall separately. Overall accuracy can go up while the system gets less safe because the model has quietly stopped abstaining.

My rough redundancy test: remove one example and rerun the eval. If the failure pattern doesn’t move, that example probably isn’t buying much.

The hard part is preventing the boundary set from becoming a museum of last month’s bugs. How are people refreshing theirs without turning every production miss into a permanent test case?

Welcome, glad you found us. This is a community for developers building with LLMs, AI agents, and all the stuff that goes around them. Whether you are shipping something to thousands of people, tinkering on a side project late at night, or just getting started and figuring it out as you go, you are in the right place.
Ask your questions, show what you built, share the thing that broke and how you fixed it, and drop theguides and tricks that helped you. This place is run by developers for developers, so the only things weask are simple: keep it useful, keep it honest, and grounded in real work. Jump into the comments and tell us what you are building right now.