u/Overall_Judge8086

I built hybrid retrieval over 790 clinical PDFs: local FAISS + BM25 merged with Google's Gemini File Search, plus re-ranking layers. I test it with 60 LLM-generated questions, each written from one page, so the correct paper and page are known. Each question runs 3 times, because File Search gives different results each run.

Overall, my system beats File Search alone:

	MRR	Right paper in top 10	Right page
File Search alone	0.56	61%	70%
My system	0.67	83%	94%

Last week I found two problems using 20 of the questions:

1. 64% of File Search results had no page number, so I now locate each chunk's text in my copy of the document.
2. My synonym-expanded query was hurting the search, so I now search with the plain question ( use expanded query for score boosting).

On those 20 questions, MRR went from 0.59 to 0.72. On the 40 other questions, which I kept aside, it went from 0.66 to 0.65: 4 improved, 3 got worse, 33 didn't change. Top-3 accuracy did rise, from 68% to 75%.

Questions:

• Is this overfitting, or are 40 questions just too few to show a difference?
• My answer model reads the top 5–8 papers anyway. Should I track hit@3/hit@5 rather than MRR?
• What's a better way to build a retrieval test set than one LLM question per page?