We blamed the model but retrieval was giving it the same paragraph four times
We blamed our model for weeks because our RAG assistant kept giving confident policy answers that missed one exception clause. Too much chunk overlap meant retrieval kept pulling near copies of the same paragraph and the exception never made it into context. We were paying extra tokens to make the wrong evidence look unanimous.
We inspected the retrieved chunks inside Braintrust traces and compared chunking runs on the failed queries. Embedding similarity showed why basic top k kept selecting copies. Adding deduplication helped, then reranking against the full question pulled the exception clause above the repeated policy text. Citation precision and groundedness both improved when the evidence set stopped repeating itself.
Those failed queries also became regression cases for us. We now score retrieval coverage separately from answer groundedness, because a model cannot cite a clause it never received (yes, obvious in hindsight). Token spend also fell because the context carried fewer duplicate passages. Not really the problem we thought we were fixing, but I'll take it.