u/Worldline_AI

Not judging, I am in this with everyone else. We read the diff, understand roughly 70% of what we see. The other 30% looks plausible. Tests pass. Merge.

What we are not doing: checking what the agent actually did during the session beyond the PR diff. How many files it read. What commands it ran. Whether it touched anything outside the stated task.

I did a quick count on my own setup:

• Sessions run this month: somewhere around 40
• Sessions where I pulled the full log: 2

The ratio is horrible, but prolly not unusual. The part I keep coming back to: we built code review culture specifically because it looking right is not the same as it being right. Right? Adding agents in the mix changed the speed, but not the reason. The diff is still not a session audit.

At some point the vibe check comes due.

Task: refactor the auth module. Agent ran 34 minutes, 51 logged actions. PR diff showed 46. Hence 5 actions never appeared in the review surface.

The five: a file modification adding an undeclared key, a package version bump, a debug log written to a folder, reads on three files outside the task scope including two from the billing module, and a shell command not in the original task plan. The PR looked clean, but nobody pulled the session trace before merging.

This is not a "broken agent" problem. The agent completed the task. But the diff is the output surface, not the evidence layer. The session trace is the evidence. Most review workflows review the output. Almost nobody looks at the full session record.

From a Sonar survey this year: 96% of developers don't fully trust AI output, but only 48% always verify before committing. The other 52% are trusting the diff.

The uncomfortable part is not that 5 actions went unreviewed. It is that you cannot be certain it was the first time. You just happened to look this once.

Last month, a cursor agent running Claude Opus 4.6 deleted PocketOS entire production database and all backups. Nine seconds, one API call. The agent had explicit rules in its system prompt: "NEVER run destructive commands unless explicitly asked." It somehow found a railway API token in an unrelated file and used it anyway.

When questioned afterward it wrote: "I violated every principle I was given. I guessed instead of verifying. I ran a destructive action without being asked. I didn't understand what I was doing before doing it."

That is a complete failure log. It names exactly what went wrong, in the right sequence too. The problem is that most teams only see this record after something breaks. The rules were in place. The agent ignored them. That gap between the rule and the actual behavior is not visible in normal output review. You see the output, ie the deleted database, but you do not see the decision chain that produced it.

The agent confessed this time. The next one might not.

The VentureBeat writeup on the coding agent exploits is a fun read, not for the security patches but for the pattern it reveals:

Different attackers, different codebases, different entry points, one shape: an agent held a credential, executed an action, and authenticated to a production system. No human session anchored the request, nobody audited the trace.

An agent acting on your behalf should never hold more privileges than you do. And even after you scope the credentials correctly, you are still making trust decisions without session-level evidence. You accepted the output. You did not look at what the agent authenticated to, what commands it ran, or when it deviated from the intended path.

The security story got loud because credentials leaked. The quieter (scarier?) version plays out every day: agents ship code, engineers approve diffs, nobody pulls the session trace.

I've been running the same task through the same Claude Code instance across several sessions this week. Different days, different context states. The outputs are meaningfully different.

Not wrong vs. right. More like: one pass took careful, incremental steps with explicit file checks before each write. Another went faster, made assumptions, and produced code that worked but had three undocumented behaviors. Both cleared CI.

The problem isn't that one was bad. The problem is I have no principled way to choose which one to ship. I'm doing it by feel: the pass that "looks more careful." That is not a system.

We have solid tooling for evaluating outputs: tests, linters, code review. We have basically nothing for evaluating the decision pattern an agent used to get there. Two different behavioral profiles, same output shape, no way to distinguish them without replaying the session manually.

Not asking about eval benchmarks or leaderboard scores. Those are population-level signals. I mean per-instance, per-run variance: does this specific agent instance, in this specific codebase context, tend to make the kind of decisions I can sign off on?

Curious what patterns people have found that persist beyond a single session.

There's a pattern I keep seeing in agent discussions lately: someone reports their coding agent "got worse" over a few weeks. The replies split into two camps: "yes, model updates broke it" vs. "you're imagining it, the model is the same." Both camps are missing the actual thing.

The model probably is the same. But the agent instance you're running today is not the same as the one from six weeks ago, different context window contents, different session history, different harness configuration, small accumulated decisions that compound. Same model. Different behavior. And you have no baseline to compare against because you never measured the first one.

This is the structural problem with how we're deploying coding agents right now: the model name is treated as the unit of measurement. "We use Claude Code" or "we switched to Codex" as if the model name tells you something about what that specific agent did in your monorepo over the last sprint.

It doesn't. Two engineers running the same model on the same codebase, with different harness setups and different session patterns, are running different agents. When one of those instances "gets worse," the right question is not "did the model change?" It's: what changed in this instance's behavior profile, and how would you know?

The engineers having the clearest picture of this are the ones keeping records at the instance level. Not "Claude Code is good at refactoring" but "this instance, on this codebase, over these 30 sessions, here is where it earned trust and here is where it didn't."

How are you currently tracking behavioral drift across agent sessions?

The live complaint about coding agents this month is context loss. Every session burns time rediscovering the repo. Switching between Claude Code, Codex, Cursor resets everything. Token costs balloon before the actual work starts. The pain is real and the threads are not exaggerating.

What the complaint stops short of is the second-order version of the same problem.

If the agent has no persistent memory of your repo, you almost certainly have no persistent record of the agent either. Each session ends, the trace evaporates, and the only thing left is the diff and your memory of how it felt to work with that instance today. Next session, you start from scratch. Not just on context. On evidence.

Concretely: you assigned a refactor to your agent two weeks ago. It went well. You routed another refactor to it last week. That one had problems you caught in review. Can you say, from a record, what was different between the two sessions? Where the agent made different decisions? Whether the second session was an off day or the beginning of a pattern?

Most teams I have talked to cannot. The agent reports completion. The PR ships or doesn't. The session is gone. The next routing call gets made on a feeling.

The reason context loss feels so expensive is that you are paying twice. Once for the agent to rediscover the repo. Again, more quietly, for the team to rediscover whether this instance is the one to trust on this kind of work.

This is becoming visible in concrete ways. Claude Code just shipped a /goal mode that runs async until a condition is met. The Mythos scan found a real curl vulnerability that the maintainer then verified. Both are signals that the agent is doing more, less observed. The record question gets louder the longer you are not watching.

The interesting question is not how to give the agent more memory. It is: what would you keep, per session, if you were going to build a record of each agent instance that actually informed the next routing decision? Decisions made, scope respected, places it pushed back, places it did not. The kind of thing that, six sessions in, would tell you something the model card never will.

There's a governance gap stat making the rounds this week: 72% of firms are in production with agentic AI, 60% have no formal governance in place.

Most of the discussion treats this as a policy problem, org charts, risk frameworks, sign-off procedures. That's not wrong, but I think it's the wrong layer to start at.

The layer underneath the policy question is this: can your team actually answer, for any given coding agent instance you're running, what that instance has demonstrated it can be trusted to do? Not "what is this model good at" in the general sense. What has this specific instance, running in your environment, on your codebase, shown it can handle reliably, and what has it consistently gotten wrong?

Most teams I've talked to can't answer that. The routing decisions are based on whoever used the agent last, what they remember working, and occasionally a benchmark rank that says nothing about performance in your specific context. That's not governance. It's informed guessing.

The evidence that would actually support a governance decision, ie session traces, behavioral data per instance, scores across dimensions like reasoning quality, constraint compliance, and handling ambiguity, most teams aren't capturing it. You get the output. The session disappears.

So you end up with a team that's in production with agents but couldn't reconstruct, for any critical deployment that went wrong, what the agent actually did step by step and whether it behaved consistently with prior sessions.

For those running agents, how are you handling this? Are you capturing session-level data, or operating on output and vibes?

There's a finding circulating this week that deserves more attention than it's getting.

The claim, backed by multiple builders comparing setups: the same model can produce a 30 to 50 percentage point performance difference depending on which harness wraps it. Claude Code versus OpenHands versus a homegrown loop, same weights, materially different results on the same task.

Most teams I talk to still pick their coding agent by model name. "We use Sonnet." "We switched to Qwen 35b." The implicit assumption is that the model is the primary variable.

But if harness design accounts for a 30 to 50 point swing, the model name is a footnote. The real question is: what did this specific agent instance, in this specific configuration, on this specific codebase, actually do in this session?

That question is almost impossible to answer from output alone. The agent's claimed output tells you what it says it did. It doesn't tell you what it reasoned, what it silently skipped, which compliance decisions it made, or whether the efficiency of this run will hold on the next one.

I've started thinking about this less as a model-selection problem and more as an instance-measurement problem. The harness matters. The codebase context matters. The specific session behavior of this instance, accumulated over time, matters more than the benchmark rank.

Genuine question for anyone building seriously with local agents: do you have any way to measure what an agent instance actually did, beyond reading the diff and hoping CI catches the rest? What does your verification layer look like?