Analyse massive documents with AI
Hi everyone,
I work at a large telecom company in the US. When we win a tender, the client sends us a requirements specification, and we write technical deliverables based on it and on our internal technical documentation (100k+ documents). A single tender can involve thousands of deliverables, some of them up to ~600 pages long.
Goal: before delivery, automatically verify each deliverable. That means checking that the facts it contains are accurate against our technical documentation, and detecting anomalies (errors, internal contradictions, inconsistencies between deliverables, duplicates).
Current plan:
Fact extraction: process each deliverable in chunks with an LLM and extract atomic facts, with their source location.
Classification: group the facts into families (measurements, architecture, calculations, frequencies, standards, quantities).
Verification: check each fact against the documentation base using RAG.
Anomaly detection: one LLM call per fact category to detect duplicates, contradictions, etc.
My questions:
- Is there a recommended approach for extracting precise facts from very long documents? How do you handle context that spans chunks ?
- How do you make sure extraction is complete (no missed facts) and doesn't produce hallucinated facts?
- For anomaly detection, does one LLM call per category scale when a category can contain thousands of facts? Would you normalize facts into a structured schema (entity / attribute / value / unit) and do part of the comparison deterministically in code rather than with an LLM?
- For RAG verification on highly technical content (part numbers, frequencies, references to standards), did hybrid search (BM25 + embeddings) or reranking make a big difference for you?
- Any experience comparing long-context models (1M tokens) with chunked extraction for this kind of task?
Any feedback, papers, tools or lessons learned would be much appreciated. Happy to share what we learn along the way.
Thanks