What breaks first when RAG moves from demo PDFs to production documents?
okay I have observed every RAG pipeline demo looks amazing on clean single-column PDFs. Then you throw it at actual customer docs like scanned forms, multi-column statements, contracts with tables nested inside tables and suddenly your extraction layer just starts lying to you. quietly. No errors, no warnings, just wrong.
things that keep breaking on me:
- tables just... dissolve: cells flatten into some soup of unstructured text, or worse, they misalign, and now values are sitting in the wrong row/column like nothing happened. Retrieval says "working fine!" the answer's just wrong lol
- headings get orphaned: chunking rips the heading away from its own content so you retrieve this floating paragraph with zero clue what it's even about
- reading order goes feral: multi-column layout gets read left-to-right straight across the page instead of per-column, so sentences are scrambled before the chunker even gets a shot at it
- figures? gone. charts, stamps, signatures, poof! and sometimes the actual answer you needed was sitting in that figure, not the text around it
anyway if you're running this in prod, which one has caused you the most rework downstream? my money's on broken tables bc it fails silent instead of loud (you don't even know it's wrong until someone complains). but reading order might just generate more garbage chunks overall, even if each one's less catastrophic.