u/Cheap_Salamander3584

"At first I thought the reports were just noise because every prompt change was going through the same eval suite and passing. If quality had actually regressed, surely the eval would've caught it. That's literally what it's there for.

Eventually I started comparing the eval cases against actual production traces instead of the outputs.

Turns out they barely looked alike anymore.

The eval set had been written months earlier around the kinds of inputs we expected users to send. It wasn't a bad dataset either. It just slowly stopped matching reality. Production had drifted into messier prompts, more ambiguous requests, weird combinations of asks, edge cases we'd never thought to include. The agent still handled the old distribution pretty well. It just wasn't seeing that distribution anymore.

Looking back, the annoying part is the green check actually made us more confident shipping prompt changes. We kept thinking ""nothing broke"" because the benchmark never moved, while production had already moved somewhere else.

We've started pulling real production traces back into the eval set every so often instead of treating it like something you build once. We use OrqAI for evals now, so feeding traces back into the dataset is fairly painless, but I don't think the tooling is really the point. It feels more like eval sets have to evolve with production or they slowly become benchmarks for a product you shipped six months ago.

The part I still haven't figured out is multi-turn conversations.

Most eval frameworks still feel very request-response oriented. Our worst failures usually aren't one bad answer. They're five or six reasonable answers that collectively take the conversation somewhere dumb. Every individual turn looks fine if you inspect it on its own.

We're still opening traces and trying to spot the moment things started drifting.

Curious how other teams deal with this. Are you continuously refreshing your eval sets from production traffic, or has anyone actually found a decent way to evaluate conversation trajectories instead of individual responses?"

Spent two days chasing what i thought was a model regression.

turned out we just had three different versions of the same prompt running.

the weird part was only one team was complaining. everyone else said outputs looked normal.

at first i ignored that because i figured if the model had changed it'd hit everyone eventually. rewrote the prompt a few different ways, reran evals, everything still passed. which just made it more confusing because if quality had actually dropped you'd expect the evals to start failing too.

finally pulled traces from the team complaining and compared them to everyone else's.

different prompt.

not massively different either. just missing one instruction.

dug a little further and found out someone had hotfixed staging months ago for that team's data. meant to be temporary. never made it back into the repo. meanwhile another small edit had gone into prod later for something completely unrelated.

so we somehow ended up with three versions.

repo.

prod.

staging.

and the only people hitting the oldest one were the team that opened the support ticket.

kind of embarrassing because nobody really did anything wrong. we just didnt have one place that answered the question "what prompt is actually live right now?"

after that we stopped keeping prompts in the codebase. moved prompt management into OrqAI mainly because i got tired of playing git detective every time something looked off. at least now i can see exactly which version is deployed where instead of guessing which branch or environment someone edited six months ago.

curious if anyone has a good way of catching prompt drift automatically.

not model drift. prompt drift.

feels like this should be one of those problems everybody solved years ago, but i dont think i've actually seen a clean approach.