u/Hungry-Horror-7577

Spent a day fixing my RAG's faithfulness! Turns out the bug was in my judge!

Setup: local RAG over books. I use an LLM as a judge to score faithfulness. Is the answer actually in the retrieved context? It breaks each answer into claims and checks them.

The judge kept flagging up to 14% of claims as "contradicted" per book. That reads as real hallucination. So I went and read the flagged cases by hand.

None of them were contradictions.

Two bugs, both in the judge:

1. Truncation. The judge only got the first 600 chars of each chunk. On some questions the supporting line was past char 600. Judge never saw it, called it a contradiction.
2. No guard. I also had a hard string-match that could confirm a quote was word-for-word in the chunk. The soft LLM judge overruled it anyway.

Fixed both. Bumped the context cap, and let the hard match win over the soft verdict! "Contradicted" dropped from 14% to about 1% across all books NICE!! 🐼 Correctness didn't move (~93%). So it was never the RAG. It was the ruler I was measuring with.

Bonus I had also built a "cite mode" where the answere must quote sources verbatim. Ran it A/B. It barely moved faithfulness, because the truncation fix had already done the work. But it did cut the padding! correct claims that weren't actually grounded in the text dropped a lot. So cite mode does help, just not where I expected. Nice lesson: before you fix the RAG, check if your evaluator is lying to you.

How do you all keep your judge honest? Do you actually read the flagged cases, or trust the number?

Running a local RAG eval over ~26 dense technical books — lots of formulas, tables, exact numbers and parameter values (the kind of content where copying a figure wrong is a real failure). Strix Halo, 128GB, all Ollama, fully offline. Two tiers: retrieval (objective) and LLM-as-judge.

Retrieval is solved — Recall@8 100%, MRR ~0.98. The judge tier is where I'm unsure.

My judge is llama3.3:70b-q8, deliberately a different family than my answerer (qwen3.5:122b) to avoid self-bias. Averages across 4 books, ~80 questions:

Correctness: ~91%
Relevance: ~89%
Faithfulness: ~60%
Hallucination rate: ~10%

Faithfulness is my problem child. But here's what's bugging me: correctness 91% next to faithfulness 60% doesn't add up — you can't be 91% correct while inventing 40% of your claims. So I suspect it's either the model padding answers with unsupported detail, or my judge being too strict when it splits answers into atomic claims.

Questions for people doing this locally:

1. Have you actually measured your judge against your own hand-labels (Cohen's kappa), or do you just trust it? Mine is unvalidated so far.
2. Is a reasoning judge (DeepSeek-R1-distill) or Llama 4 meaningfully better at catching real hallucinations than llama3.3?
3. What faithfulness range do you consider "good" for a local setup?

Happy to share config. Not selling anything, just comparing notes.

Running a local RAG eval over ~26 dense technical books — lots of formulas, tables, exact numbers and parameter values (the kind of content where copying a figure wrong is a real failure). Strix Halo, 128GB, all Ollama, fully offline. Two tiers: retrieval (objective) and LLM-as-judge.

Retrieval is solved — Recall@8 100%, MRR ~0.98. The judge tier is where I'm unsure.

My judge is llama3.3:70b-q8, deliberately a different family than my answerer (qwen3.5:122b) to avoid self-bias. Averages across 4 books, ~80 questions:

Correctness: ~91%
Relevance: ~89%
Faithfulness: ~60%
Hallucination rate: ~10%

Faithfulness is my problem child. But here's what's bugging me: correctness 91% next to faithfulness 60% doesn't add up — you can't be 91% correct while inventing 40% of your claims. So I suspect it's either the model padding answers with unsupported detail, or my judge being too strict when it splits answers into atomic claims.

Questions for people doing this locally:

1. Have you actually measured your judge against your own hand-labels (Cohen's kappa), or do you just trust it? Mine is unvalidated so far.
2. Is a reasoning judge (DeepSeek-R1-distill) or Llama 4 meaningfully better at catching real hallucinations than llama3.3?
3. What faithfulness range do you consider "good" for a local setup?

Happy to share config. Not selling anything, just comparing notes.

Been heads-down building a fully offline RAG over my trading book collection and figured I'd share where I'm at, partly to compare notes and partly because retrieval turned out to be the easy part while faithfulness is the thing eating my evenings.

The box is a Corsair AI Workstation 300 (AMD Strix Halo, 128GB unified memory, roughly 119GB usable for models). Ubuntu, everything runs through Ollama, nothing touches the cloud. The whole reason I built it this way is keeping private documents private, which matters a lot where I live (EU, GDPR).

Corpus is about 26 scanned trading books. Pipeline is Qdrant with hybrid RRF retrieval and a cross-encoder reranker on top, plus the usual extras (contextual retrieval, HyDE, a GraphRAG layer, RAPTOR, and ColQwen for the page-image side).

Models, all local through Ollama:

Answerer: qwen3.5:122b-a10b (MoE, around 87GB loaded)

Embeddings: qwen3-embedding:8b-fp16

Reranker: Qwen3-Reranker-8B at F16

Judge (only during eval): llama3.3:70b-instruct-q8. I picked a different model family than the answerer on purpose, so the judge isn't grading its own house style and quietly inflating the scores.

The eval is a gold-set I built by hand. 80 questions total, 20 per book across 4 books, and I deliberately mixed in questions the books just don't answer, to see whether the model admits it instead of inventing something. Two tiers:

Tier 1 is pure retrieval, objective, no LLM judging. Recall@8 came out at 100% and MRR at 0.889, so the right chunk is basically always in the top 8 and usually near the top. Retrieval is not my problem.

Tier 2 is the LLM-as-judge part: faithfulness, correctness, relevance, hallucination rate. Faithfulness is the weak one, sitting around 58% in an earlier run, meaning the answer sometimes slips in things that aren't actually in the retrieved notes.

I just rewrote the answer prompt based on recent faithfulness work: cite the source for every claim, copy numbers and dates exactly instead of paraphrasing them, refuse outright when the notes don't cover it, drop hedge words like "usually" or "typically", and keep answers short, since longer answers seem to drift back into the model's own training instead of the documents. Running the full 80 overnight to get a clean before-and-after on that prompt change.

Where I could really use input from people who've done this locally:

1. Judge model. Is llama3.3 70B a reasonable local faithfulness judge, or would a reasoning model like DeepSeek-R1-distill-70B, or a purpose-built evaluator like Prometheus 2, give meaningfully better agreement with a human? Has anyone here actually measured judge agreement (Cohen's kappa) against their own hand labels rather than just trusting the judge?

2. Faithfulness itself. Past the prompt, what actually moved your numbers? Claim-level verification after generation, different chunking, pulling in the parent section around a hit?

3. Sample size. Is 80 questions enough to trust the gap between two runs, or am I going to end up reading noise as signal?

Building this in the open. Happy to post the full configs and the real numbers once tonight's run finishes.

Been heads-down building a fully offline RAG over my trading book collection and figured I'd share where I'm at, partly to compare notes and partly because retrieval turned out to be the easy part while faithfulness is the thing eating my evenings.

The box is a Corsair AI Workstation 300 (AMD Strix Halo, 128GB unified memory, roughly 119GB usable for models). Ubuntu, everything runs through Ollama, nothing touches the cloud. The whole reason I built it this way is keeping private documents private, which matters a lot where I live (EU, GDPR).

Corpus is about 26 scanned trading books. Pipeline is Qdrant with hybrid RRF retrieval and a cross-encoder reranker on top, plus the usual extras (contextual retrieval, HyDE, a GraphRAG layer, RAPTOR, and ColQwen for the page-image side).

Models, all local through Ollama:

Answerer: qwen3.5:122b-a10b (MoE, around 87GB loaded)

Embeddings: qwen3-embedding:8b-fp16

Reranker: Qwen3-Reranker-8B at F16

Judge (only during eval): llama3.3:70b-instruct-q8. I picked a different model family than the answerer on purpose, so the judge isn't grading its own house style and quietly inflating the scores.

The eval is a gold-set I built by hand. 80 questions total, 20 per book across 4 books, and I deliberately mixed in questions the books just don't answer, to see whether the model admits it instead of inventing something. Two tiers:

Tier 1 is pure retrieval, objective, no LLM judging. Recall@8 came out at 100% and MRR at 0.889, so the right chunk is basically always in the top 8 and usually near the top. Retrieval is not my problem.

Tier 2 is the LLM-as-judge part: faithfulness, correctness, relevance, hallucination rate. Faithfulness is the weak one, sitting around 58% in an earlier run, meaning the answer sometimes slips in things that aren't actually in the retrieved notes.

I just rewrote the answer prompt based on recent faithfulness work: cite the source for every claim, copy numbers and dates exactly instead of paraphrasing them, refuse outright when the notes don't cover it, drop hedge words like "usually" or "typically", and keep answers short, since longer answers seem to drift back into the model's own training instead of the documents. Running the full 80 overnight to get a clean before-and-after on that prompt change.

Where I could really use input from people who've done this locally:

1. Judge model. Is llama3.3 70B a reasonable local faithfulness judge, or would a reasoning model like DeepSeek-R1-distill-70B, or a purpose-built evaluator like Prometheus 2, give meaningfully better agreement with a human? Has anyone here actually measured judge agreement (Cohen's kappa) against their own hand labels rather than just trusting the judge?

2. Faithfulness itself. Past the prompt, what actually moved your numbers? Claim-level verification after generation, different chunking, pulling in the parent section around a hit?

3. Sample size. Is 80 questions enough to trust the gap between two runs, or am I going to end up reading noise as signal?

Building this in the open. Happy to post the full configs and the real numbers once tonight's run finishes.

Okay so. A couple months ago I got tired of paying OpenAI rent every month just to have it read my own documents back to me. You know that exact moment — you're staring at the invoice going "wait… those are MY PDFs?" 😭 So I saved up and grabbed a Corsair AI Workstation 300. Strix Halo, 128GB unified memory. One rule, non-negotiable: nothing leaves the box. My data, my machine, my problem. 🔒

First real project: an offline RAG over my books. Built completely from scratch. No LangChain, no LlamaIndex, no "pip install someone-else's-opinions." Just me, Ollama, and a deeply unhealthy amount of coffee ☕😵

The unified memory is the part that actually made me weak. I've got a 122B model sitting in memory, embedder right next to it, reranker right next to that. All loaded at once. No VRAM tetris. Anyone who's ever played "unload this to load that — oh wait now I need the first one again" just felt that sentence in their spine 💀😂

Was any of it smooth? No. Not even slightly. The AMD/ROCm side fought me like I owed it money 😩 I lost entire evenings to things that "should just work" (don't lie, you have too 👀). At one point it was 2:30am, I was staring at an error the internet swore couldn't exist, and I was straight up begging my terminal like it was an abandoned pet 🥲🙏

Then I finally built an eval set, and oh boy. 📊 Recall@8 sitting at a glorious 100%, MRR 0.94, correctness 93% — and I'm feeling like an absolute genius… until I scroll down to faithfulness: 58% 💀 Turns out 100% recall means nothing when your model confidently answers from stuff it learned in pretraining instead of, you know, the actual book I handed it 😭 The retrieval is flawless. The model just refuses to read the assignment and freestyles anyway. Relatable, honestly.

But. Every single time the thing pulls the right answer straight out of one of my own books, I forgive it instantly. Completely toxic relationship. Zero regrets. Would do it again. 😌💔

So — r/RAG, two questions:

• What are you running on your Strix Halo / unified-memory rigs? 👀
• What would you point this thing at next? Pretty sure I've barely scratched the surface and I'm having way too much fun finding out 🚀

Saved up and grabbed a Corsair AI Workstation 300 a couple months back. Strix Halo, 128GB unified memory. I'd been eyeing a proper local box forever and finally talked myself into it — because paying OpenAI rent every month to read my own documents back to me started feeling a little stupid 😅

First real project: an offline RAG over my book collection. Built from scratch, no frameworks, all through Ollama. Nothing leaves the machine — that was the one rule I refused to break. My data, my box, my problem. The way it should be 🔒

The unified memory is the part that genuinely sold me. I keep a 122B-class model sitting in memory right next to the embedding and reranker models, all at once, no VRAM juggling. If you've ever played the "unload this to load that, oh wait now reload the first one" game, you know exactly the pain I'm escaping 😂 Watching the whole pipeline run local without it still feels a little unreal.

Now, was it all smooth? Absolutely not 💀 The AMD/ROCm side fought me like it owed me money. Lost a few evenings to stuff that "should just work" (we've all been there, don't lie). But every time it actually answers a question straight out of one of my books, I forgive it instantly 🥲

Curious what the rest of you are running on Strix Halo / unified-memory rigs — and what you'd point this thing at next. Pretty sure I've barely scratched what it can do, and I'm having way too much fun finding out.

Two months ago I picked up a Corsair AI Workstation 300 (AMD Strix Halo, 128GB unified memory). Honest reason: I love this stuff, and I read a lot of books — I wanted a way to reach the knowledge in them and pull exactly what I need, fast. Already-read ones and new ones.

One rule from day one: fully offline. No cloud, no APIs, nothing leaves the machine. Everything local through Ollama.

Plan was "something basic to get started." That is… not what happened 😅 Went straight down the rabbit hole — hybrid retrieval, reranking, an actual eval set so I can measure instead of guess.

Started with ~26 books. Far from done, learning as I go. Happy to share the setup — curious what the AMD/local crowd would've done differently.