The "which model is best" debate stopped mattering and most builders haven't noticed
Here's the thing I keep running into. Half my feed is people arguing about which model won this week. The other half quietly swapped to a free open-weight model months ago and built something around it that the benchmark crowd can't touch.
The benchmark is a snapshot of one model on one task. What actually compounds is the system you wrap around it.
We learned this the boring way at the ai agents company I work for. Our early agents were basically good prompts. Vee for brand voice, Ines for Instagram, a few others. They worked until they didn't, and the moment we scaled, the same prompt produced generic output for every customer because a prompt knows nothing about who it's working for. No amount of prompt polishing fixes that.
The model is the cheap part now
This is where the open-weight shift actually bites. When a free model running 300 agents in parallel beats a paid model 5x its price on real research work, the smart money stops asking "which model" and starts asking "what runs on top of it."
Three questions matter more than the leaderboard:
How many runs can you afford to throw away? Cheap tokens change what you attempt.
Who checks the output? An open swarm is confident and under-cites, so we put one strong model at a single verify gate whose only job is to refute the work, never to praise it.
Does run fifty know anything run one didn't? If every run starts from zero, you have a prompt wearing a costume. A real loop saves what worked as a reusable skill and bakes every caught mistake into a constraints file the next run reads automatically.
That last one is the whole game. The model doesn't retrain between your runs. Your skill library grows. A competitor can clone your prompt in a minute, they can't clone six months of your real runs.
Where I'll admit I'm wrong-ish
Benchmarks aren't useless. If a new open model genuinely jumps a tier on reasoning, that flows straight into the loop and every run gets better for free. So I do watch them. I just stopped treating the top line as the decision.
And the open-weight stuff is rough in real use. Ours over-cites, contradicts itself across sub-agents now and then, and needs that verify gate babysitting it. The gate costs real tokens and it's the part of the setup I'm least happy about.
So genuine question for the people deep in this: is anyone running a fully open-weight loop, verify model included, with no paid model anywhere in the chain? Every setup I trust still has one expensive model guarding the gate, and I want to know if that's actually droppable yet or if we're all quietly paying the same tax.