
Do zero-LLM health metrics predict where bugs land? Ran it across 21 repos, here's what held up
Most "AI reviews your PR" tools annoy me. They hallucinate nits, the output changes every run, and you can't tell if a flag means anything. So I wanted to know if you can get a useful review signal the boring way. Static metrics plus git history, without any LLM in the loop, so same input gives the same output every time.
The question I actually wanted answered: do deterministic code-health metrics predict which files get bug-fixed later?
How I set it up :
- Score every file at a historical commit (T0).
- Count bug-fixing commits to that file over the next 6 months.
- Correlate the T0 score with the bug-fixes that came after. No file sees its own future.
Metrics are the usual suspects plus churn signals. McCabe complexity, deep nesting, LCOM4 cohesion, god classes, clone detection, function-level churn, code age, ownership spread, change entropy. 25 of them, combined into a 1-10 per file. I ran it across 21 OSS repos, 9 languages.
What came out:
- Mean ROC AUC 0.74 at picking the files that go on to get bug-fixes. Up to 0.90 on some repos, weaker on others.
- It survives controlling for file size (partial Spearman -0.16). So it's not just "big files have more bugs," which was my first worry.
- Out-ranks churn alone by about +0.10 AUC, and prior-defect history by +0.12.
- Held on an external dataset I never touched (PROMISE/jEdit), AUC ~0.77.
I ran it head to head against CodeScene too, same 2,770 files, same commit, same labels, since it's the closest established tool. Discrimination was close (0.73 vs 0.71). The gap showed up on effort-aware ranking. Under a "you only have time to review 20% of the changed lines" budget, the deterministic score surfaced more of the real defects (recall 0.17 vs 0.07, Popt 0.61 vs 0.46). Similar at telling risky from safe, better at ordering what to look at first.
Where it falls down, because 0.74 is not magic:
- It tells you where bugs cluster, not what the bug is.
- Ranks files, doesn't read them so not a review replacement.
- Defect labels come from bug-fix commits, which is a noisy heuristic. Some "fixes" aren't, some bugs never got a clean fix commit. That ceiling is baked in.
- The 6-month window is a choice. Move it and the numbers move.
- The CodeScene edge is specifically ranking under a budget. On raw "is this file risky, yes or no," they're close.
Where I landed: the signal is real and it's reproducible, which is the part I care about for PR gating but it won't catch a logic bug in a 4-line diff.
It's good at "this PR touches a file that's been a problem for a year with one owner, slow down." Different job than line-by-line review, and I think it's the job the deterministic approach is actually good at.
I have also added agent provenance to the same which allows it to determine if an AI PR is worse than human
If anyone wants to take a look at the repo: https://github.com/repowise-dev/repowise
Also, if you have any feedback on the metrics itself or if I can try some new metric, would love to try that