r/AIQuality

I'm new to building AI agents and working on a CI-review agent: it takes a failing CI run and tries to find the root cause.

I want to start with a baseline version, then iterate on it — but I need a way to measure whether a new version is actually outperforming the old one.

Right now the agent maintains several hypotheses about the failure, assigns each a probability, and updates those probabilities as it gathers more information. Depending on its confidence, it either outputs a summary of the likely root cause, or escalates to a human developer if uncertainty is too high.

Two questions:

1. How do you evaluate an agent like this, where the output isn't a single "correct" answer but a probability distribution over hypotheses?
2. How do you get a labeled dataset of CI failures with known root causes, so I can score the agent's probability estimates against ground truth instead of just eyeballing whether the output "feels right"?

It’s been something of a mantra for a while now: use a frontier model for planning, then delegate the implementation to low-cost models, and finally verify the implementation using the frontier model (and then ask the cheap one to do the corrections)

I’m not saying it isn’t the best approach, but speaking from my own experience… I have my doubts.

Let’s say GPT-Sol Medium (or Terra High) are my planner models, and something like DeepSeek V4 Flash or Gemini 3.7 Flash handles the implementations. What I’ve noticed with this workflow is that the number of correction rounds resulting from the review is often enormous, no matter how clear the plan is. In this scenario, I end up burning ‘expensive’ tokens on planning and the review rounds, whilst also burning cheap tokens (which also add to the cost) through constant implementation and correction, which ultimately amounts to a waste of time and money.

So now I’m trying a different approach: once the planning agent has defined the plan, why change it? That agent already has the context to implement it in a single shot and, if anything, I have the implementation reviewed by a couple of low-cost agents. I know it sounds counterintuitive, and I don’t have any large-scale data to back up this approach either, but my feeling is that it works better for me.

I’ve been digging deeper into evals and AI release workflows, and there’s one part I’m especially interested in.
Teams can already compare prompts/models on quality, latency, and other eval metrics.
But I’m curious how people handle the tradeoff between quality and cost.
For example, suppose a new model:
improves task success from 85% to 90%
but doubles the cost per request
Is that a good change?
The answer probably depends on the actual customer outcome, not just the eval score or token cost individually.
I’m experimenting with comparing a baseline and candidate on the same test set, then looking at cost per successful outcome rather than cost per request.
The goal is to answer something closer to:
“Did this change improve the product enough to justify what it costs?”
For people running LLM features in production, how are you making this decision today?
Is this already part of your eval pipeline, handled manually, or mostly monitored after deployment?

I wanted to test whether price predicts writing quality on a task where "sounds human" is everything, so I built a small pipeline on Evaligo building a profile for a dating site.

The test set was 30 intentionally bad bios - the "gym, tacos, dogs" one-liner, the "partner in crime, fluent in sarcasm" special, one all-emoji, one bitter rant. A separate model (not one of the contestants) scored every output 0–1 against a rubric: reuses the person's real details, no clichés, the three tones actually differ, tip is specific not generic. Minus 0.5 if it invented facts. The judge never saw which model wrote what.

Results over 30 bios each:

Model	score	Flawless bios	$ / profile	Latency
gpt-5.6-luna	0.86	17/30	$0.0006	4.6s
gemini-3.5-flash	0.79	7/30	$0.0163	8.4s
claude-haiku-4-5	0.67	4/30	$0.0024	5.5s

The interesting failures:

• Haiku's worst scores all came from the cliché-stuffed bios — it echoed "partner in crime" energy back instead of replacing it.
• Flash wrote well but its witty versions drifted "a bit forced" per the judge, and 27x the cost was mostly invisible thinking tokens.
• Luna's only recurring weakness: the three tones occasionally blurred together. It never invented facts.

Also learned the hard way that sample size matters: at n=6 flash and haiku tied, at n=30 they clearly separated. The whole benchmark cost about $0.90.

Happy to share the rubric or the raw score distributions if anyone wants to poke at the method. (Disclosure: I build the tool I ran this on, so questions about the harness are fair game too.)

How can we give people more agency when interacting with artificial intelligence systems? That is one of the questions my research aims to answer. 

I am an assistant professor in the University of Maryland Department of Computer Science, where I am leading a new research effort to test whether the reasoning processes used by advanced AI systems will remain transparent. At the center of my research is a widely used technique known as chain-of-thought reasoning, in which AI models generate step-by-step explanations of how they reach their answers. 

Feel free to ask me about AI transparency, natural language processing and more. I’ll be answering questions on Wednesday, August 12, from 11 a.m. to 1 p.m. EDT (15-17 UT).

Bio: Sarah Wiegreffe is an assistant professor in the Department of Computer Science at the University of Maryland, College Park (UMD). She is a member of the CLIP (Computational Linguistics and Information Processing) lab and also affiliated with UMD’s AI Interdisciplinary Institute (AIM) and Institute for Advanced Computer Study (UMIACS).

Sarah works on the explainability and interpretability of deep learning systems for language, with a focus on understanding how language models make predictions to make them more reliable, safe, and transparent to human users. She has been honored as a three-time Rising Star in EECS, Machine Learning, and Generative AI. She was previously a postdoc at the Allen Institute for AI and the University of Washington and, before that, received her Ph.D. and M.S. degrees from Georgia Tech.

Other links:

• Google Scholar
• Personal Website
• Recent coverage: UMD's Sarah Wiegreffe Receives Grant to Stress-Test the Future of AI Transparency

Username: /u/umd-science

https://preview.redd.it/2xp45odb1wih1.jpg?width=5000&format=pjpg&auto=webp&s=dcded40e0bd97b6920dbaba3d6bea5e9e40f158d

Built RFP Agent AI - automates the 2-5 day manual tender quoting process for wire/cable manufacturers into 10 seconds using a 5-agent pipeline (PDF parsing → SKU matching → LME pricing → risk analysis → quote generation).

Known problems I've already found:

• Catalog only has ~6 SKUs, real world needs 5000+
• Vector search giving 46% match scores - wrong approach?
• No live LME API yet
• No human checkpoint on low confidence matches

Is multi-agent overkill here? Is vector search right for structured spec matching? What would you do differently?

GitHub: https://github.com/AnshMNSoni/B2B-RFP-Agent.git

Would love brutal honest feedback - not looking for encouragement, looking for loopholes, wrong assumptions, and better architectural approaches. If you've built something similar or work in procurement/manufacturing tech, your perspective would be especially valuable.

https://preview.redd.it/inihdtzot6ih1.png?width=1151&format=png&auto=webp&s=ee76aa903c3dd30d8e6e46de2fa85d018e1645bd