u/Ruca_AI

I’ve been thinking a lot about false confidence in LLM/agent debugging tools.

When two agent runs differ, it’s tempting for an observability tool to pick the first suspicious-looking span and present it as “the cause.”

But in a lot of cases, the structural evidence is ambiguous:

- repeated tool calls can be hard to align safely

- some captured content may be unavailable or redacted

- traces may be incomplete

- later differences do not necessarily imply causality

I’ve been experimenting with a different rule in a local-first run-comparison tool I’m building:

If the evidence is not sufficient to establish an investigation starting point, return `uncertain` instead of selecting the most plausible-looking location.

The tradeoff is obvious: you sometimes give the user less information.

But the benefit is that the tool does not silently convert ambiguity into confidence.

I currently test this with a deterministic regression corpus where ambiguous cases are expected to fail closed rather than guess.

I’m curious how people working on LLM observability think about this:

Would you rather have a debugging tool return a weaker/uncertain result more often, or make a best-effort guess as long as it explains why?

And what kinds of evidence would you personally require before calling something an actual “root cause” rather than just an investigation starting point?

Hey everyone,

I’ve been building an open-source project called TraceMotive.

The basic idea is:

Given two AI agent executions, TraceMotive compares their observed behavior, finds the first supported divergence, and lets you jump into the evidence around it.

It runs locally.

The goal is not to claim root cause or automatically explain why something happened. One of the design principles I care about most is that if the structural evidence is ambiguous, TraceMotive should say that the result is uncertain instead of guessing.

I just released v0.5.0.

This release was mostly about making the project more adoptable rather than adding a huge new feature.

Some of the work in v0.5:

- packaged `tracemotive serve` and `tracemotive demo`

- structured JSON diff support

- Safe Later Observations for additional supported evidence

- improved first-time-user onboarding

- Python 3.10 / 3.12 CI

- frontend test/build CI

- dependency auditing and Dependabot

- threat-model/security documentation

- explicit compatibility, limits, and storage docs

- clean wheel/sdist installation dogfooding

- a 30-scenario evidence-conservative regression corpus

For that regression corpus, the current results still have:

- false-confident meaningful divergence: 0

- false-confident investigation starting point: 0

There are still intentional limitations.

For example:

- LangGraph is not currently supported.

- The validated OpenAI Agents SDK range is `>=0.17,<0.18`.

- TraceMotive does not claim RCA, causal inference, confidence scoring, reconvergence, or recovery detection.

A bit of context: I’m a high-school student, and I built the first version after roughly a week of programming experience, heavily using AI coding tools.

I know that’s an unusual way to start an OSS project, so I’ve been trying to compensate by being strict about tests, failure cases, compatibility claims, and not claiming more than the evidence supports.

At this point, the thing I need most isn’t another feature idea — it’s real users.

If you build AI agents and have a run you could try this on, I’d really appreciate feedback about:

- where installation/onboarding feels confusing

- whether the comparison is actually useful

- cases where TraceMotive becomes uncertain

- agent execution patterns the current model handles badly

Thanks to everyone who gave feedback on the earlier versions — several of those comments directly influenced v0.5.

I was invited to share TraceMotive here earlier, and I just released v0.4.0.

TraceMotive is a local-first OSS tool for comparing AI agent executions and helping answer:

“Where did these two runs first diverge in observed behavior?”

The main change in v0.4.0 is that the comparison result is now organized as an investigation workflow rather than just a trace diff:

• Look here — the first evidence-supported place to investigate
• What changed — conservative structured JSON differences
• Evidence — what was actually observed
• Next — direct navigation to the corresponding left/right spans
• What TraceMotive does not know — explicit limitations and uncertainty

Other additions in v0.4.0:

• Minimal investigation cockpit
• Conservative structured JSON diff
• Direct left/right span navigation
• Additive /api/v4 comparison contract
• First-run onboarding
• Deterministic identified and uncertain demo scenarios
• Fresh-checkout / installed-wheel E2E validation
• PyPI Trusted Publishing

One design constraint I’m intentionally keeping is that TraceMotive does not treat the first observed divergence as proof of root cause.

If traces are incomplete, capture is unavailable/redacted, or repeated spans cannot be safely aligned, the result should remain uncertain rather than force a match.

In the current 30-scenario adversarial corpus:

• 15/15 expected confident behavioral-divergence cases were identified
• 14/14 supported investigation starting points were identified
• 0 false-confident behavioral-divergence results
• 0 false-confident investigation-starting-point results

Those numbers are corpus-scoped, not a universal accuracy claim.

Install:

pip install "tracemotive[server]==0.4.0"

I’m a high-school student building and maintaining this with heavy use of AI coding tools.

I’d especially appreciate feedback from people working on LLM observability about the investigation flow, uncertainty handling, and whether this kind of “first supported divergence” view would actually save time in real debugging.

I just released TraceMotive v0.4.0.

TraceMotive is a local-first OSS tool for comparing AI agent executions and helping answer:

“Where did these two runs first diverge in observed behavior?”

v0.4.0 focuses less on adding more tracing data and more on turning an existing comparison into an investigation workflow.

New in this release:

• Minimal investigation cockpit
• Conservative structured JSON diffs
• Direct left/right span navigation
• Additive /api/v4 comparison contract
• First-run onboarding
• Deterministic identified and uncertain demo scenarios
• Fresh-checkout / installed-wheel E2E validation
• PyPI Trusted Publishing

The investigation flow is now roughly:

Look here → What changed → Evidence → Next → What TraceMotive does not know

One constraint I’m intentionally keeping:

TraceMotive does not claim that the first observed divergence caused the later failure. If the evidence is ambiguous or incomplete, it should remain uncertain instead of force-matching spans or inventing an explanation.

In the current 30-scenario adversarial corpus:

• 15/15 expected confident behavioral-divergence cases were identified
• 14/14 supported investigation starting points were identified
• 0 false-confident behavioral-divergence results
• 0 false-confident investigation-starting-point results

Those numbers are corpus-scoped, not a universal accuracy claim.

Install:

pip install "tracemotive[server]==0.4.0"

I’m a high-school student building and maintaining this with heavy use of AI coding tools, so I’m learning a lot while working on it.

I’d especially appreciate feedback on whether the new investigation workflow actually saves time when debugging real agent runs.

Hey everyone,

I just released v0.3.0 of TraceMotive, an open-source local-first debugger for AI agent executions.

The problem I'm trying to solve is pretty simple:

You have one agent run that worked and another that failed or behaved differently.

Most tracing tools can show you both executions, but you still have to manually inspect the traces and figure out where they started behaving differently.

In v0.3, TraceMotive can compare the two runs and identify the first behavioral divergence that is actually supported by the structural evidence.

The workflow is roughly:

good run

vs

bad run

→ deterministic structural alignment

→ first supported behavioral divergence

→ diagnostic findings

→ investigation starting point

→ additional observations / context / uncertainty

Some examples of findings:

- tool input changed

- tool output changed

- new error observed

- error resolved

- tool added / removed

- execution subtree changed

- tool repetition changed

- model / request parameter / trace status changes as context

A big design goal was avoiding fake certainty.

TraceMotive does NOT claim that the first divergence caused the later failure.

If repeated tool calls can't be safely aligned, content is redacted, capture is unavailable, or the trace is incomplete, the result can explicitly be `uncertain`.

Everything remains local-first.

No TraceMotive telemetry.

Collector/UI remain loopback-only.

Captured data is treated as untrusted.

There's also now a deterministic demo that doesn't require an API key:

pip install tracemotive==0.3.0

tracemotive serve

Then in another terminal:

tracemotive demo

That generates a reference run and changed run and opens the investigation view.

I'm still early and would especially appreciate feedback on:

- whether this investigation workflow is actually useful

- cases where the structural alignment is too conservative

- agent frameworks you'd want supported next

I'm a high-school student building this with heavy use of AI coding tools, so I'm also learning a lot while building it.

Would love to hear what breaks :)

A few days ago I released TraceMotive, a local-first tracing/debugging tool for AI agent execution.

The first version worked, but the feedback was pretty clear:

- traces disappeared after restart

- setup required multiple terminals

- users needed Node/npm for the UI

- it behaved more like a trace viewer than a debugger

So I used that feedback as the scope for v0.2.0.

The new release adds:

- persistent local SQLite storage

- `tracemotive serve` for one-command startup

- a packaged production UI with no Node/npm runtime requirement

- trace-to-trace comparison

- Changed only filtering

- side-by-side field differences

- explicit ambiguous/unavailable states instead of guessing repeated tool-call pairings

One thing I found while testing comparison was that naive ordinal matching can confidently pair the wrong repeated tool calls after insertion/removal/reordering. I ended up treating those groups as ambiguous instead of pretending the pairing is exact.

I’m mainly looking for real failure cases now. If you work with agent traces, what would still stop you from using something like this?