u/Nimsumdimsum

If you're building a document-processing pipeline today, does it actually make sense to send every stage through the same high-end multimodal model? My instinct is that a lot of document work doesn't need the most capable model.

For example:

• Clean PDFs / straightforward OCR: traditional OCR, direct text extraction, or a lightweight model may be enough.
• Parsing and simple extraction: a faster, lower-cost model such as Gemini Flash-class models may handle this well.
• Handwriting, poor scans, complex tables, or ambiguous fields: this may be where you route to a more capable multimodal/reasoning model.

The part I'm unsure about is whether the accuracy and cost advantage of model routing is actually worth the orchestration complexity in production.

Here’s how I’m thinking about the trade-offs:

• Accuracy: One model gives you more consistent behavior, but it may be overkill for simple documents and weaker on certain edge cases. Multi-model routing lets you optimize by document type or task, but poor routing decisions can hurt accuracy.
• Latency: One model means fewer routing steps and simpler execution. Multiple models can keep easy documents on faster models, but retries and escalations may add latency.
• Cost: One model is easier to predict, but expensive if a premium model handles everything. Routing can reduce cost significantly if most documents can stay on lightweight models.
• Privacy: One provider/model can simplify governance and data handling. Multiple providers add complexity, although routing could also keep sensitive documents on private or internally hosted models.
• Fallback behavior: With one model, a retry may simply reproduce the same failure. With routing, low-confidence outputs can escalate to another model or eventually to human review.
• Maintenance: One model is much easier to operate. Multi-model pipelines require more evals, routing logic, monitoring, version management, and regression testing.

I'm especially interested in the fallback strategy.

Would you use:

small model → larger model → different provider → human review

or simply:

one strong model → human review when confidence is low?

And what would you use as the routing signal: OCR confidence, image quality, handwriting detection, document type, extraction confidence, schema validation failure, or something else?

For anyone running document AI at meaningful volume: has multi-model routing actually reduced cost and improved accuracy, or does the added complexity outweigh the benefit?

okay I have observed every RAG pipeline demo looks amazing on clean single-column PDFs. Then you throw it at actual customer docs like scanned forms, multi-column statements, contracts with tables nested inside tables and suddenly your extraction layer just starts lying to you. quietly. No errors, no warnings, just wrong.

things that keep breaking on me:

- tables just... dissolve: cells flatten into some soup of unstructured text, or worse, they misalign, and now values are sitting in the wrong row/column like nothing happened. Retrieval says "working fine!" the answer's just wrong lol

- headings get orphaned: chunking rips the heading away from its own content so you retrieve this floating paragraph with zero clue what it's even about

- reading order goes feral: multi-column layout gets read left-to-right straight across the page instead of per-column, so sentences are scrambled before the chunker even gets a shot at it

- figures? gone. charts, stamps, signatures, poof! and sometimes the actual answer you needed was sitting in that figure, not the text around it

anyway if you're running this in prod, which one has caused you the most rework downstream? my money's on broken tables bc it fails silent instead of loud (you don't even know it's wrong until someone complains). but reading order might just generate more garbage chunks overall, even if each one's less catastrophic.