u/ringarc

Hardware: RTX 5070 Ti 16 GB with 128 GB DDR5. I built llama.cpp from source. The model was Qwen3.8-27B, a dense hybrid DeltaNet + attention model. My original setup used UD-Q4_K_XL at 17.9 GB. It couldn't fit into 16 GB of VRAM, so I used partial offload with -ngl 44 and left the remaining layers in system RAM. Decode speed was 6.75 tok/s. I decided it wasn't practical and switched to a 35B MoE using expert offload with --n-cpu-moe. That model reaches 66 tok/s. A comment on internet prompted me to test another setup. I used UD-Q3_K_XL, which is 13.4 GB. It's an Unsloth dynamic quant, with sensitive tensors kept at 4 to 8 bit while most of the model uses 3 bit. The settings were -ngl 99, -fa on, -ctk q8_0 -ctv q8_0. A 64k context still didn't fit beside the resident weights because the compute buffer ran out of memory. A 32k context worked with -ub 512. These are the decode speeds in tok/s at 500 / 4k / 16k tokens of context: - Spilled 27B UD-Q4: 6.75 at every tier because system RAM bandwidth is the limit - Resident 27B UD-Q3: 51.8 / 51.2 / 48. Prefill was 378 tok/s, with 9 s TTFT at 16k. Total usage was 14.7 GB. - 35B-A3B MoE with --n-cpu-moe 28 and the same FA + KV q8 settings: 66.4 / 65.1 / 63.4. It used 12.1 GB. On the MoE, FA + KV q8 reduced memory use by 1.2 GB without changing speed. I now enable those settings by default. For testing quality, I used a private agentic coding band with 22 tasks. The target is a FastAPI + React + Postgres + Mongo app. It includes bug fixes, feature changes, new features, a migration, a performance fix, and one intentionally impossible specification. Each model gets a shell inside a docker box and up to 40 steps. Hidden tests determine the score. The model must also submit a final "what did you do" report, which is verified against git and the real test runs. These results come from one trial per model, so they're only indicative: - Resident 27B UD-Q3: mean 0.49, with 9/22 perfect - 35B MoE: 0.56, with 10/22 perfect - gpt-oss:20b: 0.47, with 6/22 perfect The 27B matched the MoE on localised debugging, with both scoring 6/7 perfect. It fell behind on multi-file feature work, scoring 1/11 against 3/11. On the larger tasks, it often spent all 40 steps reading without making an edit. Its stronger area was honesty. The 27B made one false "done" claim across 13 failures. The MoE made 4 in 11, and gpt-oss made 4 in 15. I can't separate the model difference from the cost of 3-bit quantisation. The comparison is 0.49 versus 0.56, but the 4-bit 27B was never fast enough to run this band usefully. On an earlier and easier suite, the Q4-vs-full-precision tax on this machine was about +0.02 overall. Reasoning and repo coding took the largest hit, so a bigger loss from Q3 would make sense. Here's the theory I'd like people to check. The 27-30B dense range seems designed around unified-memory Macs, where these models fit completely at Q4 or Q8. A 16 GB card can only hold them at Q3. Meanwhile, small-active-parameter MoEs such as 35B-A3B and gpt-oss-20B seem like the models actually intended for this hardware. Is that consistent with what others are finding? A few more questions: - IQ4_XS is 15.7 GB. Has anyone managed to keep a 27B IQ4_XS fully resident on 16 GB using a small context and KV q4? If so, does the quality improvement over Q3 justify losing context? - Has anyone compared 3-bit EXL3 or another importance-aware 3-bit format with Unsloth dynamic Q3 on the same 27B using coding tests rather than perplexity? - What decode speed do people target for agentic workflows? In a shell loop, 52 tok/s felt usable to me. 6.75 did not. My conclusion is to start every new dense model in this class with resident dynamic Q3 + FA + KV q8, profile it, and only then decide whether it's any good. I'd done those steps in the wrong order.

I collected ten of the viral "RAG in production" posts and read them side by side. Two things jumped out.

First, every single one presents "the problem is retrieval, not the LLM" as a contrarian insight. When every post makes the same contrarian claim, that claim is the consensus.

Second, they directly contradict each other. One lists "Graph RAG over engineering" as a top production mistake, three others sell knowledge graphs as the fix. One says start with dumb fixed-size chunks and measure, two others call fixed-size chunking mistake number one. One demands you measure everything, then offers a "Hit Rate above 70%" threshold with no data behind it. And my favorite: a whole article on how Google's Open Knowledge Format is replacing the vector database. OKF is real. It is a 450-line spec describing a folder of Markdown files with YAML frontmatter. It has one required field. It replaces nothing, because a file format cannot replace a search mechanism.

I think the confusion has a specific cause. "The RAG debate" is not one debate. It is five separate debates that all got tagged #RAG, each with evidence from a different domain:

1. Grep vs vectors. Claude Code dropped vector search for grep and it worked. An Amazon paper got a grep agent to 94.5% of a RAG pipeline's faithfulness with no vector store. But Cursor published the counter-evidence: adding a trained embedding model to their grep-using agent improved eval accuracy 12.5%, so their agent uses both. Notice all the grep-won evidence is from code, where identifiers are exact and an agent can retry. Nobody has shown it on messy enterprise PDFs. The paper behind this discourse is literally titled "Is Grep All You Need?" and its actual finding is that the agent harness matters more than the retrieval method. That nuance did not survive contact with LinkedIn.

2. RAG vs long context. Chroma's context rot report: 18 frontier models, performance degrades as input grows, on every model, long before the advertised limit. Elastic's cost comparison is where the "1,250x cheaper" number comes from ($0.00008 vs $0.10 per query), though that ratio is one corpus, one cheap model, no prompt caching, so treat it as a data point. One honest concession: if your whole corpus is a few hundred thousand tokens and rarely changes, full context plus prompt caching is a legitimate architecture, not a hack.

3. RAG vs GraphRAG. Graph extraction on a 5GB corpus reportedly went from ~$33k (early 2024) to ~$33 (mid 2025, LazyGraphRAG-style), single source so grain of salt, but the direction tracks. Cost is no longer the objection. The objection is that you are now maintaining entity resolution, an ontology, and a graph that drifts as documents change, forever, paid in engineering time. Graphs earn it on real multi-hop queries ("who approved the vendor that supplied the part that failed"). If your logs are mostly "what is our refund policy", you do not have a GraphRAG problem. Log queries for two weeks and classify them before deciding.

4. RAG vs CAG. Load the corpus once, keep the KV cache, answer from the cached state. Genuinely new, works, and the fine print is in the original paper: the entire knowledge source has to fit in the context window, and context rot applies before it is full. Real option for a bounded static corpus (product manual, policy handbook). Category error to call it a RAG replacement.

5. RAG vs memory. Agent memory is retrieval over your own past interactions, with writes. The write side is genuinely new engineering (what to keep, what to summarize, what to expire). The read side is retrieval, and the vendors' own benchmarks are recall and precision numbers.

All five debates reduce to one question: given my corpus, my queries, my freshness needs, and my cost ceiling, what is the cheapest selection mechanism that survives my failure cases?

For most document QA systems that resolves to something boring: fixed-size chunks with overlap, hybrid search (BM25 + vectors, this should be the default, not an upgrade), metadata filters, generous context on the generation side, and fifty hand-scored eval questions before buying any upgrade. The thing the listicles skip entirely: parsing. Chunking operates on whatever your parser produced, and PDFs with tables that extract as word salad kill more retrieval than any chunking choice. I suspect half the "fixed-size chunking ruined my retrieval" stories are parsing failures wearing a chunking costume.

Two admissions so you don't have to make them for me: I say "90% of systems need only the boring baseline" and that number is made up, I believe the shape but nobody has surveyed this. And every number above is someone else's measurement (Cursor's, Elastic's, Chroma's), I verified sources but have not published my own before-and-after, which is the same gap I am criticizing in the genre.

I wrote this up in more detail (including the reranking and indexing sections, and when you genuinely are in the minority that needs the advanced tier): https://ringarc.ai/labs/tech/rag-five-debates

Happy to be told which of the five debates I got wrong.

I dismissed a 27B dense model after getting 6.75 tok/s on a 16 GB card. A fully resident Q3 with flash attention + KV q8 reached 52 tok/s instead. These were the tradeoffs.

Your boring RAG pipeline is probably fine