Intern building an RFP compliance checker — RAG keeps producing false “Missing” results
I’m a CS intern working on an internal system that compares an RFP against a vendor’s technical proposal.
The current flow is roughly:
RFP
→ extract requirements
→ retrieve/rerank proposal evidence
→ LLM classifies each requirement:
Compliant / Mismatch / Missing
→ compliance score
The score was around 30%, mostly because many requirements came back Missing.
After building an independent ground truth from the original RFP and proposal, I found the issue isn’t just retrieval:
- some requirements are lost during PDF parsing/extraction
- some are split/merged incorrectly
- relevant evidence sometimes exists but doesn’t reach the final Top-K
- in other cases the evidence reaches the LLM and it still judges it incorrectly
- some requirements mix technical and financial conditions that should be checked in different documents
For one reviewed sample, many of the system’s Missing results were actually supposed to be Compliant or Mismatch.
I’m now considering a design like:
Requirement
→ split into material conditions
→ determine expected evidence source per condition
→ retrieve evidence per condition
→ verify each condition
→ deterministic final verdict
For exact things like durations, percentages, certifications, etc., I’m also thinking of using deterministic checks instead of letting the LLM decide everything.
Has anyone built something similar for contracts, RFPs, policies, or compliance?
Would you approach this as per-condition RAG, claim verification/NLI, an agentic search layer, or a hybrid deterministic + LLM system?