When RAG Works in Testing but Fails in Production
Has anyone dealt with a RAG system that performs really well during testing but starts returning irrelevant or incomplete answers in production?
We’re facing an interesting problem.
The knowledge base contains thousands of documents with overlapping information, different versions, tables, PDFs, and occasionally conflicting data. On a small test dataset, retrieval accuracy looks good. But once the document volume increases, the system sometimes retrieves a related document instead of the correct document.
The confusing part is that the LLM itself seems to be working fine. The problem appears to be somewhere between document ingestion, chunking, embeddings, retrieval, and re-ranking.
We’re considering several approaches:
• Hybrid search using vector + keyword retrieval
• Metadata-based filtering
• Better chunking strategies
• Re-ranking retrieved results
• Query rewriting before retrieval
• Adding document/version awareness
But improving one part sometimes seems to negatively affect another.
If you’ve solved a similar production RAG problem, what was the actual bottleneck?
Was it the embedding model, chunking strategy, retrieval architecture, metadata, or something else?
Would really appreciate practical suggestions or lessons learned from real implementations.