r/AIQuality

I burned through half my Fable 5 allowance in 25 minutes. The same kind of session on Opus 4.8 takes me about 2 hours to hit that mark. So the cap isn't really the story. The burn rate is. Fable 5 eats tokens so much faster that "50% weekly" translates to way fewer working hours than you'd assume when you sign up.

Which is why I think "always use the strongest model available" is the most expensive habit in AI-assisted dev right now. For me, Fable 5 is worth it for maybe 20% of my week. The trick is knowing which 20%.

Where it clearly earned its slot: complex builds where I couldn't spec things out properly. I gave it a SimCity-style simulation project with barely any detail, and the level of detailing it produced was on another level. Opus 4.8 gave me a decent result on the same thing. Fable 5 was the clear winner. The gap shows up specifically when the project is complex AND your prompt is thin. That combination is where the extra capability actually converts into saved time instead of just saved ego.

So my routing rule since the cap:

• Regular dev tasks where I have enough information to hand over — clear requirements, known codebase, defined scope — Opus 4.8, every time. The flagship buys me almost nothing there except a drained quota.
• Genuinely complex tasks where I don't have much detail to give: that's what I reserve Fable 5 for. It fills gaps better than anything else I've used.

Be honest. Most "agent eval" I see in the wild (including ours until recently) is a spreadsheet of prompts someone runs manually before big changes. That's not a regression suite. That's a vibe check with extra steps.

A real regression suite, the way we have for normal software, would mean: versioned test cases, runs automatically on every change, fails the build on regression, tracks pass-rate over time, and grows when new failure modes are found.

I want to know who's actually doing this for agents, and what it took to get there. Because the gap between "spreadsheet of prompts" and "real regression suite" feels large and I'm trying to figure out if it's worth crossing or if everyone's secretly still on spreadsheets.

"At first I thought the reports were just noise because every prompt change was going through the same eval suite and passing. If quality had actually regressed, surely the eval would've caught it. That's literally what it's there for.

Eventually I started comparing the eval cases against actual production traces instead of the outputs.

Turns out they barely looked alike anymore.

The eval set had been written months earlier around the kinds of inputs we expected users to send. It wasn't a bad dataset either. It just slowly stopped matching reality. Production had drifted into messier prompts, more ambiguous requests, weird combinations of asks, edge cases we'd never thought to include. The agent still handled the old distribution pretty well. It just wasn't seeing that distribution anymore.

Looking back, the annoying part is the green check actually made us more confident shipping prompt changes. We kept thinking ""nothing broke"" because the benchmark never moved, while production had already moved somewhere else.

We've started pulling real production traces back into the eval set every so often instead of treating it like something you build once. We use OrqAI for evals now, so feeding traces back into the dataset is fairly painless, but I don't think the tooling is really the point. It feels more like eval sets have to evolve with production or they slowly become benchmarks for a product you shipped six months ago.

The part I still haven't figured out is multi-turn conversations.

Most eval frameworks still feel very request-response oriented. Our worst failures usually aren't one bad answer. They're five or six reasonable answers that collectively take the conversation somewhere dumb. Every individual turn looks fine if you inspect it on its own.

We're still opening traces and trying to spot the moment things started drifting.

Curious how other teams deal with this. Are you continuously refreshing your eval sets from production traffic, or has anyone actually found a decent way to evaluate conversation trajectories instead of individual responses?"

Posting our current stack because the AIQuality community has been the most useful place for honest eval discussions I've found. Sharing what we run and where the gaps still are. Curious what others are using and what's actually catching production issues.

Product context: B2C multimodal AI product (text + image + voice), ~50k monthly active users, three model providers (OpenAI, Anthropic, in-house fine-tuned Llama), one customer-facing agent (support), one internal agent (analytics Q&A).

Eval stack broken out by concern:

Prompt regression (prompt or model changed, did outputs degrade)

• Tool: Promptfoo, runs in CI on every PR touching prompts
• Coverage: ~80 test cases per agent, plus prompts unit-tested against gold standards
• Catches: most prompt-tweak side effects, model-update regressions
• Gap: doesn't handle multi-turn well

Multi-turn conversation quality

• Tool: Custom LLM-as-judge with structured rubrics
• Coverage: 200 synthetic conversations per agent, regenerated monthly
• Catches: context loss, contradictions across turns, goal drift
• Gap: judge model drift requires manual recalibration when we update the judge

Adversarial behavioral testing

• Tool: TestMu's Agent to Agent Testing Cloud
• Coverage: hallucination, bias, toxicity, off-scope, prompt injection, PII leakage rubrics
• Catches: behavioral failures under adversarial pressure that our handwritten tests miss
• Gap: their out-of-the-box rubrics are great but we still maintain custom rubrics for our domain-specific compliance needs (we're in finance)

Production observability

• Tool: LangSmith for traces, our own pipeline for tool-call logging, Datadog for latency/cost
• Coverage: 100% of production conversations sampled with PII scrubbing
• Catches: real-world failure modes our pre-deployment eval misses
• Gap: lag between "production failure happens" and "we notice it"

Hallucination detection (specific because we're high-stakes)

• Tool: combination of Agent to Agent's hallucination rubric + RAGAS for retrieval-grounded scoring + custom factuality checks against our knowledge base
• Coverage: every response that cites a fact gets a factuality score
• Catches: most factual errors, especially in RAG flows
• Gap: doesn't catch hallucinations of policy/process information (e.g., agent inventing a refund policy) - we use human review for this

PII leakage and compliance

• Tool: Agent to Agent's compliance rubric + Presidio for PII scanning
• Coverage: every conversation scanned for PII patterns
• Catches: most PII leakage, including system prompt leakage attempts
• Gap: novel adversarial framings sometimes slip through

Where we still don't have a great answer:

• Long-tail evaluation. Our eval catches the top 80% of failure modes. The long tail of weird user inputs is mostly caught in production via observability, which is reactive.
• Multi-modal eval. Image and voice eval is less mature than text. We're piloting some image factuality checks but the tooling is younger.
• Cost. The full eval stack costs us maybe ~$3k/month in tool subscriptions + compute. For our scale it's justified but it adds up.

What's working for everyone else? Particularly curious about: how are people handling multi-modal eval, and how are you measuring eval ROI (because the executives ask).

We're an early-stage startup (3 engineers) and have been shipping AI features for about 6 months. Up to this point our testing has basically been me and one other engineer eyeballing outputs in staging before each release, plus whatever users report after.

I finally got time carved out this sprint to set up actual evals (been looking at Braintrust, Langfuse, Arize, etc.) and the tooling side seems pretty straightforward. What I'm stuck on is the dataset itself. So far I've hand-picked ~20 examples from our logs that cover our main use cases plus a few edge cases that have burned us before. And it honestly feels embarassingly small. Every guide I find is super vague on this. Some say start small and iterate, others are throwing around numbers in the hundreds or thousands.

Also unsure about sourcing. Pulling real inputs from production logs feels like the obvious move since it reflects what users actually do, but our logs are full of repetitive/low-effort prompts. I could write synthetic cases to fill the gaps, but then I feel like I'm just testing for stuff I already know to look for.

So for anyone who's set this up, how big was your dataset when you started with? Did you grow it over time or do a big upfront push? And what's your rough split between real production data vs synthetic?