Negative result: vector distance can't tell "weak evidence" from "no evidence", and here's the data that convinced me
I built an eval harness for a document QA pipeline. It answers security questionnaires from a company's own policy docs. 24 questions, labels written down before the system was ever run against them, three deterministic passes.
It scores 15 out of 24. Nine failures. Six of them share one cause, and I want to talk about the fix I couldn't make.
The setup. Answers are gated on how far the best retrieved chunk sits from the question. The cutoff is 0.3. Below it the system answers, above it it abstains. Six of the nine failures are questions where the model produced a correct, well hedged, properly cited answer that the gate then threw away.
The obvious fix. Raise the cutoff. Those six sit at 0.323, 0.340, 0.359, 0.384 and 0.412.
Why I couldn't. One question that has to abstain sits at 0.321. Its evidence genuinely doesn't support an answer, and it only abstains correctly because 0.321 is above 0.3. Every failure I'd want to rescue needs a cutoff higher than that.
There's no value that recovers any of the six without also flipping a correctly abstaining question into confidently answering something its evidence doesn't support. My eval treats that as disqualifying no matter what it does to the total, so I logged it as no change made.
What I think is going on. Distance measures how close the nearest thing is. I was asking it whether there's evidence here at all. Those two come apart, and at this corpus size there's no clean place to draw the line. It isn't miscalibrated, it's the wrong signal.
Two things the harness caught me on, both by instrumenting instead of assuming:
First, I'd logged one question as retrieving cleanly at rank 1, because something came back from the right document. When I actually read what got retrieved, the top hit was a completely different section and the real evidence was down at rank 4.
Second, I'd logged three failures as the model seeing the evidence and abstaining anyway, and I had a prompt fix planned. When I instrumented the actual confidence values, the model had answered correctly every time and the gate was discarding it afterwards. There was no prompt bug. A NOT_FOUND status collapses two different causes into one visible outcome, and only reading the underlying values tells them apart.
What I'm actually asking. Has anyone found a confidence signal that separates these properly? I'm considering a cross encoder reranker score instead of raw distance, an entailment check between the answer and the passage it cited, or looking at agreement across several retrieved chunks. I'd rather hear what's worked on a real corpus than what a paper claims.
Harness, labels and every tuning pass including the rejected ones are here, and the threshold data is in EVAL.md: