u/nefayran

A few days ago I wrote up a benchmark where a code knowledge graph over MCP (codebase-memory-mcp) didn't make a coding agent cheaper than plain grep. That run was small: 24 questions, repos up to 25k graph nodes, no Opus. So I ran a bigger one and committed the plan before the first run. Disclosure: the graph goes through cbm-lean, a small wrapper around that server that I wrote.

Setup: GitHub Copilot CLI 1.0.90, headless, one question per session. Two arms: view, grep and glob only, or the same plus the graph and a routing rule (graph for callers and call chains, grep for exact strings). Four public repos: the FastAPI full-stack template (1.6k nodes), dub (25k), kubernetes (146k), vscode (219k). 16 questions per repo, half structural (callers, callees, call chains, what breaks if a signature changes), half exact (a config value, an env var, the line of an error or a definition). Haiku 4.5 three times per question per arm, sonnet 5 twice, opus 5.5 once on 12 structural questions. 664 runs, every answer checked against keys I verified by hand.

Tokens per correct answer on the structural questions, graph vs grep:

kubernetes and vscode
  haiku    48k vs 209k
  sonnet   29k vs 40k
  opus     25k vs 27k
small FastAPI repo
  haiku    33k vs 26k
  sonnet   19k vs 10k

On the big repos, per question, haiku with the graph spent 0.29 of what it spent with grep and was cheaper on 15 of 16 questions. It's the only result that survives the Holm correction. On exact questions the graph made no measurable difference for haiku, and sonnet paid 13 to 42% more for it.

Why haiku gains so much: with grep it needed a median of 22 model calls per answer on kubernetes, and one question took 114. With the graph, 5. Every call resends the conversation, so long searches get expensive fast. Sonnet took 8 calls with grep and 4 with the graph, opus 4 or 5 either way. My guess, which I didn't test directly: a graph replaces search turns, so it pays when the model would otherwise take many of them.

For Copilot users specifically: premium requests are charged per prompt, however many calls the agent makes, so inside Copilot the graph saves time rather than money. Haiku's structural answers on the big repos took a median of 71 s with the graph and 157 s with grep. On the small repo the graph arm was slower.

Three failures I didn't expect, in case you're wiring a graph into an agent:

• The graph's snippet has "lines": 43 next to "start_line": 93. Both haiku and sonnet answered horizontal.go:43 for a struct that starts on line 93 and is 43 lines long. My wrapper now renames the field to line_count.
• No call edge for a bare call to a function imported with from x import y. Asked what recover_password calls, the graph arm said "only get_user_by_email" 5 times out of 5, without a single grep. It calls three more.
• Copilot's grep tool shows line numbers only with "-n": true. Haiku passed n, no dash, in 920 of its 1,949 grep calls, got no numbers and counted lines itself. 78 of its 79 wrong exact answers are a wrong line number. Sonnet passed -n in 508 of 571 calls.

Caveats: one harness and one graph, 8 questions per cell on the small and medium repos, and opus answered 12 questions once each.

Write-up: https://allkeep.org/en/lab/when-a-code-graph-pays-off Plan, questions, driver and all the runs: https://github.com/nefayran/code-graph-vs-grep

If anyone has run something similar with Claude Code or another graph, I'd like to compare.