Looking for advice on designing hard coding tasks for SOTA models
​
I’m working on creating coding tasks for evaluating SOTA coding models, and I’m trying to understand how experienced people design problems that are actually challenging without being unfair.
The main things I’m struggling with are getting the reference solution naturally above the required LOC threshold and keeping the model pass rate around 50% or lower.
How do you usually come up with tasks that are difficult enough for strong coding models without artificially adding complexity or unnecessary code?
Do certain types of changes work better, such as deeper feature additions, tricky edge cases, or changes that require modifying multiple parts of a codebase?
Also, how do you estimate difficulty before running the models? I’m trying to avoid wasting evaluation runs on tasks that end up being too easy or fail because the problem itself is unclear.
Would appreciate any practical advice from people who have experience designing SWE-bench-style or other coding benchmark tasks.