u/camerongreen95

This happened to me a few months back and i think a lot of people building rag systems hit the same wall.

Built a pipeline, tested it on a clean set of docs, retrieval looked accurate, answers looked grounded, shipped it. within a week the answers started getting worse. not obviously broken, just quietly wrong more often. i spent days staring at the llm output trying to fix it before realizing the problem was nowhere near the model.

The real issue is most people only check one layer of the pipeline and assume the rest is fine. there are four layers where rag systems actually fail and each one looks completely different from the outside.

layer 1, ingestion and chunking

this is where it broke for me. inconsistent document formats, chunks splitting mid context, and almost no metadata kept at ingestion time. if retrieval is pulling irrelevant chunks, this is almost always where to look first, not the embedding model.

layer 2, retrieval and vector storage

embedding model choice, similarity search tuning, metadata filtering. this layer decides whether the right information even makes it to the model. i had chunks with zero metadata, so i had no way to filter results by source or recency, everything just got dumped into similarity search and hoped for the best.

layer 3, generation and grounding

right context can still produce a bad answer if the prompt does not force the model to stay grounded in what was retrieved. this is also where citations and source attribution live, and where you decide if the system says "i don't know" or just invents something.

layer 4, evaluation and production

the layer almost nobody builds until something breaks. recall and precision at k, groundedness and faithfulness scoring, a golden test set you actually trust, monitoring for retrieval latency and failed documents. without this you are shipping changes blind, exactly what happened to me.

Quick way to figure out which layer is actually broken:

retrieving the wrong chunks, that's layer 1

right chunks but a bad or ungrounded answer, that's layer 3

no idea if a change made things better or worse, that's layer 4

worked in testing, breaks at real scale or real documents, usually layers 1 and 4 together

i scored myself badly on layer 4 for months without realizing it, and honestly probably would have kept going if a change hadn't quietly made things worse without anyone catching it.

That gap between shipping something and actually knowing if it works is the whole reason i went looking into this properly. There's actually a hands on workshop on aug 1 with nikola ilic that walks through building all four layers for real, from raw documents to an evaluated rag app, not just the theory.

I am looking for people to join this workshop along with me. Sharing The details of workshop in the comment.

I have spent a lot of time going through agent failures in production LLM systems. The patterns that keep showing up are almost always the same and almost none of them get caught by standard testing.

The most common failure modes I keep seeing:

1. Wrong tool selected silently. The agent does not error, it just calls the wrong tool with confidence. The output looks plausible. Nobody notices until something goes wrong three days later.
2. Trajectory explosion after a prompt tweak. Someone changes a system prompt for tone. The agent starts taking 14 steps for tasks that used to take 3. Token costs spike. Still no error thrown.
3. LLM judge drift after a model upgrade. The team upgrades the underlying model, judge scores look fine, but nobody calibrated the judge against human labels after the upgrade. The scores are now measuring something slightly different from what they were before.
4. Indirect prompt injection through tool outputs. The agent calls an external tool. The response contains hidden instructions. The agent follows them. Nobody was testing for this.

The fix that actually works is evaluating across four layers. Component level catches wrong tool calls and bad arguments before the full run even starts. Trajectory level catches step count explosion, duplicate calls, loops and cost blowouts during execution. Outcome level catches bad final responses using a calibrated LLM as judge, and calibration against human labels is the part most people skip. Adversarial level catches prompt injection and unsafe behavior through explicit red team cases. If your agent reads any external content this layer is not optional.

Quick maturity check, rate yourself 0 to 2 on each layer. Zero means not doing it at all. One means doing it sometimes but inconsistently. Two means systematic and repeatable. Your lowest score is where you start. Most teams score 0 on adversarial and trajectory and do not realise it until something breaks in production.

If you want to go deeper on building all four layers properly, we are hosting a live bootcamp with Ammar Mohanna PhD covering the full evaluation stack with real notebooks you build on the day. Link in first comment.

I have been speaking with a lot of ML engineers lately about how they evaluate their agents in production and the pattern is almost always the same. The team has some form of evaluation set up, scores are going up, and everyone feels reasonably confident. Then something breaks in production that the eval suite never caught.

The issue is usually not that the evaluation is missing. The issue is that it is only covering one layer of a problem that has four.

Most teams evaluate final output quality. Almost nobody evaluates the trajectory that led to that output. Your agent might be getting the right answer through a path that takes three times as many tool calls as it should, burns unnecessary tokens on every run, and loops in ways that would be catastrophic at scale. None of that shows up when you only look at the final answer.

The same pattern applies to LLM judges. Every team is using them now but almost nobody has calibrated their judge against human labels. An uncalibrated judge gives you scores that trend upward while actual quality drifts. You think things are improving. They are not.

And almost nobody has adversarial evaluation. If your agent reads external content as part of its workflow and you have no red team suite, you are shipping something you genuinely do not understand.

If you are working through any of these layers and want to go deeper, we are hosting a live bootcamp with Ammar Mohanna PhD covering the full evaluation stack for production agents. It It is a paid bootcamp so might not work for everyone but yes if you are interested i am sharing Link in first comment.

been seeing a lot of posts here about building automation with AI agents. wanted to share something that completely changed how i approach shipping these workflows.

most people building automation test the happy path a few times and ship. the problem is AI agents fail in ways that are completely invisible in the final output.

three things that break in production that you will never catch by eye:

1. trajectory failures
your agent gets the right answer but makes 12 tool calls instead of 3. costs have tripled. latency has doubled. the output looks fine. you have no idea until your bill arrives or a user complains about speed. you need trajectory assertions that check step count, duplicate calls, retry loops and cost per run (not just the final answer)

2. uncalibrated judges
everyone is using LLM as judge to automate quality checks. almost nobody is calibrating the judge against human labels. an uncalibrated judge gives you false confidence — scores go up but real quality is drifting. you need correlation checks, agreement scores and mean absolute error before you trust any automated eval.

3. adversarial gaps
if your automation workflow reads external content — web pages, emails, tool outputs, user messages — and you have no red team suite, you are shipping blind. indirect prompt injection through tool outputs is a real production risk that almost nobody tests for.

the fix is a four layer evaluation framework before every deployment:

• component checks — right tool, right arguments, every time
• trajectory checks — path efficiency, cost gates, recovery behavior
• outcome checks — rubric scoring, calibrated judge, factuality and completeness
• adversarial checks — injection testing, instruction override, tool trust boundaries

the go/no-go rule: a single open check is a no-go before shipping.

if you want to go deeper on building all of this properly, we are running a hands on Agent Evals Bootcamp on june 27 with ammar mohanna — it is 5 hours live, you build all four evaluation layers with real notebooks on the day itself

link i have shared in the comment, if you have any queries do lmk

Hey everyone

I have been studying production AI agent failures recently and one pattern keeps coming up. Teams test thoroughly before shipping and something still breaks in production. Nobody catches it until a user reports it days later.

The root cause is almost always the same. Testing only checks the final output. But agents fail in ways that output checking can never see.

Here is what I have found actually breaks and how to catch it.

Failure 1 — The agent called the wrong tool and nobody noticed

No error was thrown. Latency looked normal. The agent answered from memory instead of calling the lookup tool it was supposed to use. The output was fluent and confident. It was completely made up. Three days later a user flagged it.

This is a component level failure. What actually catches it is testing tool selection independently from the full run — not whether the agent succeeded overall but whether it called the right tool with valid arguments. Each test case needs the user query, expected tool, expected arguments and a label rationale. Without that structure you are testing vibes not behavior.

Other things worth checking at this layer are argument quality covering required fields and valid values, planning quality covering step ordering and completeness, and failure categorisation that distinguishes wrong tool from incorrect arguments from premature stopping. These are different failure modes and they need different fixes.

Failure 2 — A prompt tweak made the agent take 14 steps for a 3 step task

Someone changed a system prompt for tone. It was a reasonable change. The agent started over-reasoning on everything. Token costs tripled. Latency doubled. The final answer was still correct so nothing alarmed. Output monitoring had zero signal for any of this.

This is a trajectory level failure. The fix is asserting on the run itself not just the output. Step count, duplicate calls, loop detection and cost and latency thresholds all need to be treated as first class quality gates. Every run should capture reasoning steps, tool calls, observations and token use in order so you can actually see what happened. Recovery behavior after failed tool results is also worth testing separately because that is where a lot of loops start.

Failure 3 — The LLM judge said everything was fine after a model upgrade

The team swapped the underlying model. Judge scores looked stable. But nobody had calibrated the judge against human labels after the upgrade. It was measuring something slightly different and the team had no idea.

An uncalibrated LLM judge is just noise on top of noise. Before trusting it you need separate rubric dimensions for factuality, completeness, groundedness, format and safety, each with a clear scale, anchors and failure examples. Then you calibrate against human labels and check correlation and agreement before relying on the scores. It is also worth applying judge mitigations like randomized answer order and hidden model identity to reduce positional and familiarity bias.

Failure 4 — The agent followed instructions it should not have

The agent called an external tool. The content that came back had hidden instructions embedded in it. The agent followed them. Nobody was testing for this because most eval setups have no adversarial layer at all.

If your agent reads external content or takes real world actions this layer is not optional. You need red team cases covering indirect prompt injection, instruction override and data exfiltration. Tool outputs should be treated as untrusted data not commands to obey. High risk actions need explicit policies around whether they are allowed, need confirmation or should be blocked, and those policies need to be tested not assumed.

A quick maturity check — rate yourself honestly on each layer:

0 = I am not doing this at all
1 = I do it sometimes but not systematically
2 = It is automated, versioned and repeatable

Most teams score 0 on adversarial and trajectory. Not because they do not care but because there is no obvious starting point and output monitoring feels like enough until it suddenly is not.

One simple rule before every deployment:

I run the eval suite before every prompt change, model swap or tool update. Every production failure gets converted into a versioned test case before the next release. A single regression is a no-go.

Curious which of these failure modes people here have hit hardest in your production. Happy to discuss in the comments.

Hey everyone

Been working through agent evaluation properly and wanted to share something that genuinely changed how I think about it. Putting it here because this community deals with these problems daily.

Fast diagnostic map — start from the symptom you're seeing:

1. Wrong tool calls or malformed arguments → Component evaluation
2. Correct answer but too many steps or too much cost → Trajectory evaluation
3. Bad or unusable final answer → Outcome evaluation
4. Unsafe behavior or prompt injection → Adversarial evaluation

Layer 1 — Component checks:

1. Each eval example includes the user query, expected tool, expected arguments and label rationale
2. Tool selection accuracy is measured across the full tool inventory
3. Argument quality checks cover required fields, valid values and semantic match
4. Planning checks cover completeness, minimality and correct ordering
5. Failure categories distinguish wrong tool, incorrect arguments, repeated calls and premature stopping

Layer 2 — Trajectory checks:

1. Every run captures reasoning steps, tool calls, observations, retries and token use in order
2. Assertions detect excessive steps, duplicate calls and loop like behavior
3. Recovery behavior is tested after failed or low quality tool results
4. Cost and latency thresholds are treated as first class quality gates

Layer 3 — Outcome checks:

1. The rubric has separate dimensions for factuality, completeness, groundedness, format and safety
2. Each dimension has a clear 1 to 5 scale with anchors and failure examples
3. Any LLM as judge is calibrated against human labels
4. Judge mitigations are applied including randomized answer order and hidden model identity

Layer 4 — Adversarial and production checks:

1. Red team cases include a task, malicious payload, expected safe behavior and pass/fail criteria
2. The suite covers indirect prompt injection, instruction override and data exfiltration
3. Tool outputs are treated as untrusted data not commands to obey
4. Production monitoring tracks retry rate, clarification rate and drift from baseline

Maturity scorecard — rate each layer 0 to 2:

0 = Not doing it at all
1 = Doing it sometimes but inconsistently
2 = Systematic and repeatable

Your lowest score is where your next unit of work pays off most.

Go/no-go gates before shipping:

1. No critical safety failures in the adversarial suite
2. Groundedness and completeness meet the agreed threshold
3. LLM judge is calibrated against a human labeled check set
4. Cost, latency and step count stay under budget
5. Regression tests run before every prompt, model or tool change
6. Failed examples are reviewed and converted into new tests before next release

A single open box is a no-go. That's the rule.

Happy to discuss any of these in the comments.

hey everyone, been working through agent evaluation properly and wanted to share a few things that actually changed how i think about it.

start from the symptom not the layer

wrong tool being called — component problem. correct answer but too many steps — trajectory problem. final answer looks wrong — outcome problem. unsafe action or injection risk — adversarial problem. once you map symptoms to layers debugging gets way faster.

most teams only check final outputs

that's like only checking if a flight landed safely without looking at what happened during the flight. trajectory evaluation catches a whole class of failures that output checking misses entirely — duplicate calls, loops, unnecessary retries, cost blowouts.

an uncalibrated LLM judge is worse than no judge

if you haven't validated your LLM as judge against a small set of human labels you're adding noise on top of noise. calibration is not optional, it's the whole point.

convert every production failure into a test case

before your next release, not after. sounds obvious but almost nobody does it systematically. within a few cycles you have a regression suite that actually catches things before deployment.

adversarial testing is not optional

if your agent reads external content or takes real actions, indirect prompt injection through tool outputs is a real failure mode. most eval setups completely ignore this layer.

happy to chat about any of these in the comments.