u/Individual_Squash_59

I built a free public benchmark that tests whether AI can catch bugs in code. Some of the bugs are real security regressions that actually shipped in open-source projects (real CVEs, put back in with the upstream fix as the answer key), and others are bugs I inject into real repos myself. Then Claude, GPT, Gemini, and DeepSeek review the diff, and I score who catches the bug, who flags fake ones, and whether they give the same answer twice. The surprising part: on the hardest bugs, some models catch them every single run while others miss every single run, so which model you pick matters more than I expected. No signup to browse, and the CVE-based ones link back to the real fix so you can check them. If you ship code with AI, I'd love you to poke holes in the method. benchmodel.io/leaderboard

I built a free public benchmark that tests whether AI can catch bugs in code. Some of the bugs are real security regressions that actually shipped in open-source projects (real CVEs, put back in with the upstream fix as the answer key), and others are bugs injected into real repos myself. Then Claude, GPT, Gemini, and DeepSeek review the diff, and I score who catches the bug, who flags fake ones, and whether they give the same answer twice. The surprising part: on the hardest bugs, some models catch them every single run while others miss every single run, so which model you pick matters more than I expected. No signup to browse, and the CVE based ones link back to the real fix so you can check them. If you ship code with AI, I'd love you to poke holes in the method. benchmodel.io/leaderboard

I built a free public benchmark that tests whether AI can catch bugs in code. Some of the bugs are real security regressions that actually shipped in open-source projects (real CVEs, put back in with the upstream fix as the answer key), and others are bugs I inject into real repos myself. Then Claude, GPT, Gemini, and DeepSeek review the diff, and I score who catches the bug, who flags fake ones, and whether they give the same answer twice. The surprising part: on the hardest bugs, some models catch them every single run while others miss every single run, so which model you pick matters more than I expected. No signup to browse, and the CVE-based ones link back to the real fix so you can check them. If you ship code with AI, I'd love you to poke holes in the method. benchmodel.io/leaderboard

I built a free public benchmark that tests whether AI can catch bugs in code. Some of the bugs are real security regressions that actually shipped in open-source projects (real CVEs, put back in with the upstream fix as the answer key), and others are bugs injected into real repos myself. Then Claude, GPT, Gemini, and DeepSeek review the diff, and I score who catches the bug, who flags fake ones, and whether they give the same answer twice. The surprising part: on the hardest bugs, some models catch them every single run while others miss every single run, so which model you pick matters more than I expected. No signup to browse, and the CVE based ones link back to the real fix so you can check them. If you ship code with AI, I'd love you to poke holes in the method. benchmodel.io/leaderboard

I keep seeing "AI code review" tools and my honest first reaction is probably yours too: an LLM reviewing LLM-written code is often slop reviewing slop. So instead of asking anyone to trust a bot, I tried to measure the thing can frontier models actually catch real, shipped bugs, and how often do they cry wolf?

What I did

• Took real historical regressions actual CVEs from Keycloak, Django, and Spring and reintroduced each one into the code, with the upstream fix as the ground-truth answer. These are bugs maintainers actually shipped and later patched, not bugs I wrote to flatter a model.
• Ran Claude, GPT-4o, Gemini, and DeepSeek over each, blind, and scored the three things these tools usually hand-wave:
  • Detection rate - did it catch the planted bug?
  • False positives - how much noise on clean diffs?
  • Consistency - same diff, same model, run twice: does it even agree with itself?

What I found (real numbers from the leaderboard)

The spread on hard regressions is brutal and binary. On the vLLM deserialization-RCE regression and the Parse Server operator-injection-account-takeover regression, Claude Sonnet and Gemini 2.5 Pro caught the bug on all 3 runs — and GPT-4o and DeepSeek missed it on all 3. Same diff, opposite outcome depending on the model.

A miss plus noise is worse than silence. On that Parse Server regression, GPT-4o and DeepSeek didn't just miss the real bug they each raised false positives on top of it, netting a negative weighted score (−8.9). The wrong model doesn't just fail to help, it actively wastes your triage time.

"Caught it" ≠ "told you it mattered." On the cross-file suite, DeepSeek flagged 100% of the planted bugs but mis-rated severity on every one real high-severity issues quietly downgraded to medium/low. Detection and calibration are separate problems, and most tools only talk about the first.

The thing that surprised me: run-to-run consistency was basically perfect across every suite, the same model on the same diff gave the same verdict three times (zero variance). The flakiness I expected wasn't the failure mode; which model you pick was. That's the whole argument for measuring it.

The part I'll defend: this isn't "trust my reviewer." It's a public, vendor-neutral leaderboard, I don't sell a "reviewer model," I score the ones you already pay for, and every result links to the real CVE and its upstream fix so you can check my work. If the method is wrong, it's wrong in public.

On the obvious objections, because I'd raise them too:

• "It's still just an LLM." Yes and the data shows they're uneven and miss real things (complex authorization logic especially). The point isn't "LLMs are great," it's "here's exactly where each one is and isn't trustworthy, measured." Using a second, independent model (not the one that wrote the code) and cross-model agreement helps, but I publish the misses too, not just the wins.
• "This replaces SonarQube / linters." It doesn't, and I won't claim it does. Static analysis is deterministic and catches a different class of thing. LLMs catch semantic/intent bugs linters miss, and miss things linters catch. Complementary, not a replacement.
• "Where does my code go?" You bring your own API key, the diff goes to your Anthropic/OpenAI/Google/DeepSeek account, reviewed in memory, not stored by me, never used for training. And I checked my own site so I'm not a hypocrite about it: no Google Analytics, no third-party trackers, no marketing pixels. The only browser storage is your theme choice and your login session.

Where I actually want holes poked:

• Is "reintroduce a historical CVE" fair, or does it leak hints a real PR wouldn't have?
• Better ways to measure false positives than what I'm doing?
• Does cross-model consensus actually cut noise in a real workflow, or just average it?

You can browse the leaderboard with no signup: https://www.benchmodel.io/leaderboard. If you want to run your own diff, paste one, or just a GitHub PR URL, though that part needs a login + your own key, which I know is friction.

Building this solo and in the open about the method. Tear the methodology apart, genuinely the most useful thing you can do for me.

I keep seeing "AI code review" tools and my honest first reaction is probably yours too: an LLM reviewing LLM-written code is often slop reviewing slop. So instead of asking anyone to trust a bot, I tried to measure the thing can frontier models actually catch real, shipped bugs, and how often do they cry wolf?

What I did

• Took real historical regressions actual CVEs from Keycloak, Django, and Spring and reintroduced each one into the code, with the upstream fix as the ground-truth answer. These are bugs maintainers actually shipped and later patched, not bugs I wrote to flatter a model.
• Ran Claude, GPT-4o, Gemini, and DeepSeek over each, blind, and scored the three things these tools usually hand-wave:
  • Detection rate - did it catch the planted bug?
  • False positives - how much noise on clean diffs?
  • Consistency - same diff, same model, run twice: does it even agree with itself?

What I found (real numbers from the leaderboard)

The spread on hard regressions is brutal and binary. On the vLLM deserialization-RCE regression and the Parse Server operator-injection-account-takeover regression, Claude Sonnet and Gemini 2.5 Pro caught the bug on all 3 runs — and GPT-4o and DeepSeek missed it on all 3. Same diff, opposite outcome depending on the model.

A miss plus noise is worse than silence. On that Parse Server regression, GPT-4o and DeepSeek didn't just miss the real bug they each raised false positives on top of it, netting a negative weighted score (−8.9). The wrong model doesn't just fail to help, it actively wastes your triage time.

"Caught it" ≠ "told you it mattered." On the cross-file suite, DeepSeek flagged 100% of the planted bugs but mis-rated severity on every one real high-severity issues quietly downgraded to medium/low. Detection and calibration are separate problems, and most tools only talk about the first.

The thing that surprised me: run-to-run consistency was basically perfect across every suite, the same model on the same diff gave the same verdict three times (zero variance). The flakiness I expected wasn't the failure mode; which model you pick was. That's the whole argument for measuring it.

The part I'll defend: this isn't "trust my reviewer." It's a public, vendor-neutral leaderboard, I don't sell a "reviewer model," I score the ones you already pay for, and every result links to the real CVE and its upstream fix so you can check my work. If the method is wrong, it's wrong in public.

On the obvious objections, because I'd raise them too:

• "It's still just an LLM." Yes and the data shows they're uneven and miss real things (complex authorization logic especially). The point isn't "LLMs are great," it's "here's exactly where each one is and isn't trustworthy, measured." Using a second, independent model (not the one that wrote the code) and cross-model agreement helps, but I publish the misses too, not just the wins.
• "This replaces SonarQube / linters." It doesn't, and I won't claim it does. Static analysis is deterministic and catches a different class of thing. LLMs catch semantic/intent bugs linters miss, and miss things linters catch. Complementary, not a replacement.
• "Where does my code go?" You bring your own API key, the diff goes to your Anthropic/OpenAI/Google/DeepSeek account, reviewed in memory, not stored by me, never used for training. And I checked my own site so I'm not a hypocrite about it: no Google Analytics, no third-party trackers, no marketing pixels. The only browser storage is your theme choice and your login session.

Where I actually want holes poked:

• Is "reintroduce a historical CVE" fair, or does it leak hints a real PR wouldn't have?
• Better ways to measure false positives than what I'm doing?
• Does cross-model consensus actually cut noise in a real workflow, or just average it?

You can browse the leaderboard with no signup: https://www.benchmodel.io/leaderboard. If you want to run your own diff, paste one, or just a GitHub PR URL, though that part needs a login + your own key, which I know is friction.

Building this solo and in the open about the method. Tear the methodology apart, genuinely the most useful thing you can do for me.