Does llms.txt actually help a model read a site? We ran a sealed 3-condition bench on 8 arms (4 local Q4, 4 frontier cloud) — and published the number that cuts against our own registered result
Hey everyone — founder of strata→signal here, a small local-first software workshop and research lab (we build what I call non-hostile AI tools: run on machines we operate, no accounts, no analytics, and every claim on the site carries receipts you can check).
The llms.txt argument is two years old and mostly receipts-free, so we tried to buy some receipts. Three conditions, same 30 sealed questions about our own estate:
- C-MAP — the model gets our llms.txt files in context (3,211 tokens)
- C-HTML — the model gets the site's own prose at an equal budget (3,088 tokens)
- C-NONE — the model gets nothing. This is the contamination meter: if an arm answers from training data, the sealed set is burned. The set was written freshness-armored; C-NONE came back ~zero across all eight arms.
The roster: four local arms on our own GPU — qwen3.8:27b, qwen3.6:27b, gemma4:26b, llama3.3:70b, all Q4_K_M — and four frontier cloud arms (glm-5.2, deepseek-v4-pro, kimi-k3, gpt-5.5). No Claude arm sits, deliberately: a Claude wrote the exhibit page, and seating one would stack a conflict on a conflict. (The judging in our other benches uses family recusal for the same reason.)
What we found, honestly, both directions: the registered reading fell 61.5% toward llms.txt — but that lead is carried by navigation questions, and our own extractor is why: the map block carried the only URLs in the room (fifty occurrences, thirty-six distinct), the HTML block carried none. Cut the navigation items — a cut we did NOT register, made after seeing the direction it moves, published as transparency rather than result — and the fact questions alone read 71.4% toward the site's own prose at the same token budget. Our one-line take: llms.txt behaved like a map, not an encyclopedia. It knows where things are; it lost on what things say. (Counts, not verdicts — n=30 on one site doesn't resolve a direction, and the page says so in italics right under the table.)
Two receipts that surprised us:
- The economics are upside-down at the full-file end. Anthropic's llms-full.txt — the "just inline everything" variant — weighs 30.7 MiB, call it eight million tokens: roughly $80 to read once at Fable 5 input rates, ~$40 at Opus 5 or GPT-5.5. That's dinner for a family, per read. Our whole estate map costs about three cents.
- In thirty days of our server logs, no AI crawler asked for our llms.txt. Not once, on any of our properties that kept logs. ClaudeBot alone made 594 requests and fetched robots.txt 161 times — and never the map. (Our logs, our month — we can't speak past them; the per-crawler table ships in the kit.)
Everything is published: the sealed golden set, every model reply verbatim, the scoring code, the API bill ($1.87 of a $4.00 pre-registered ceiling — 663 calls crossed the wire against a sealed plan of 674, and the gap is itemized), the counting rules, and the full history file (39 dated sources on how the argument actually unfolded). Kit is CC BY 4.0. Check our arithmetic.