u/Hydronix2731

I installed the two biggest debugging skills on skill.sh: Matt Pocock's diagnosing-bugs and Superpowers' systematic-debugging and tested how effective they actually were against a plain claude code prompt.

The bugs are 15 real ones from my own repo's history (yes, only one repo, but I decided against taking from benchmarks because those test cases are often contaminated). Then I graded each run with the tests from the real fix, which none of them could see.

The skills weren't worse, but did they did ~3 extra minutes per bug to get to the same place for no uplift and increased cost from having the skill installed.

All three tests successfully fixed 9/15 bugs; however they didn't all fix the same bugs. Plain Claude gave up on one that both skills solved. Meanwhile, Matt's skill missed one the others got, and Superpowers missed a 12-line fix that plain Claude did in 6 minutes for $0.79. This could possibly just be from variance, as I only ran each skill against each bug three times.

Matt Pocock's skill did provide some small uplift: it was the only one that never broke another test, and it made the smallest changes (median 32 lines vs. 41 and 48).

Reasoning:

According to research, when models already know what to do(like how to debug), skills don't help much unless they introduce a new method that the model hasn't seen already. Perhaps Anthropic trained their models to Matt Pocock and Superpowers' debugging processes, or at the very least something similar.

Conclusion:

Obviously, this doesn't mean you shouldn't install Matt Pocock or Superpowers. But if you have them installed or plan on installing them, perhaps consider uninstalling the debugging portion to save on token costs.

When skills work, they work absurdly well. Microsoft's SkillOpt has a spreadsheet skill that makes Claude go from 22 to 82 on SpreadsheetBench. NVIDIA ran with/without skills on 300+ verified skills and got ~41 points average lift on correctness. SkillsBench found curated skills add ~16 points on average, with some domains jumping 50+. The potential for skills to improve agent work has almost as much uplift as upgrading model classes.

But a lot of the time they do nothing. Vercel found the agent never retrieved the skill in 56% of cases, so even if the skill turns haiku into fable for UI work, 56% of the time it doesn't matter. Even if the skill is triggering, sometimes it doesn't do anything. I ran an evaluation of Claude's built-in /code-review, and found that compared to the prompt "review this code", code-review was only 3% better and passed only one more check(out of more than 30).

So when are skills actually useful? When skills contain something the model doesn't know inherently, like how to use an API, how your repo manages the database, or the specifics of how it should call an MCP. SkillsBench backs this up: the biggest gains were in domains like healthcare, the smallest in general software engineering, and models writing their own skills got zero benefit.

How do you tell which of your skills are being called and which ones are impactful even when called? All of the papers judge skills the same way: run a model with access to the skill, and one without, and measure the difference between the outputs on the same suite of tasks. NVIDIA's SkillEvaluator and the SkillsBench repo are good places to start; however, they do require more setup.

SkillEvaluator requirements (for the live with/without tier):

• API keys
• The agent CLI itself (Codex, Claude Code, etc.) with its own logged in credential
• A sandbox: Docker or cloud(local for experimental use)

SkillsBench requirements:

• Pre-written tasks written in their format
• An environment with skilled in the container
• uv and the BenchFlow CLI
• API keys
• A sandbox: Modal or Docker

For slightly easier use, there's an open source tool I made that takes the evaluation architecture from SkillEvaluator without any of the requirements (runs through your subscription, on your harness), and also keeps track of live triggers for whether or not the skill gets called at all. Terum

Took us 2 hours after we had won our game of testing different SAM levels to see what we could break through. 15 clicks per second over a sustained period was enough to break through.

We tried 80 for another hour but didn't get anywhere lol