u/Acne_Discord

More info: https://github.com/lechmazur/debate/

Debate Benchmark tests how well models defend a position through sustained, adversarial, multi-turn opposition across hundreds of topics. It demands broad knowledge, factual accuracy under pressure, sharp rebuttals, and arguments that hold together round after round.

Every matchup runs twice on the same motion, with PRO and CON swapped to control for side advantage. Three judges from distinct model families independently evaluate each debate’s winner and margin.

---

Profile excerpts:

Claude Fable 5.1

A forceful, flexible comparative debater whose strongest recurring traits are direct rebuttal, counterfactual framing, and explicit weighing. Judge-perceived strength is broad and statistically clear.

A highly comparative, mechanism-first debater. Blind transcripts show direct engagement in 299/302 debates, weighing in 265/302, and burden contests in 173/302.

Rebuttal is the clearest edge: 8.10 mean, +0.71 versus field.

GPT-6 Astra

Overall, Astra is a disciplined, epistemically careful rebuttal specialist whose comparative weighing lands better than its presentation.

A comparative, burden-focused debater that narrows disputes to marginal costs and benefits rather than accepting broad moral framing: “The right comparison is not privacy versus children.”

Questions often pressure-test the opponent’s strongest example, while concessions acknowledge the best opposing cost before pivoting to a narrower comparative claim.

Rhetorical effectiveness is the clearest judge-perceived weakness.

Originality was essentially field-average.

GLM-5.3

A mechanism-first, burden-conscious debater with especially strong rebuttal, rhetoric, and originality. Its recurring edge is converting concrete details into direct clash and comparative weighing.

GLM-5.3 (high) debates by fixing the burden early, interrogating mechanisms, and then weighing practical consequences. Blind transcripts showed question-type behavior in 192/196 debates, direct engagement in 190/196, weighing in 168/196, and strategic concessions in 121/196. A characteristic framing is: “The proposition is not "four-day weeks are nice." It is that the state should mandate them, by statute, at zero reduction in base pay.”

Muse Spark 1.3

A reliably organized, adaptable, rhetorically forceful debater whose clearest measured advantage lies in presentation and originality.

A compact, question-driven debater: questions appeared in 291/292 blind transcripts, direct engagement in 285/292, and explicit answer forms in 276/292. Weighing was also common (264/292), often contrasting reversibility, scale, and permanence: “Relaxation can be devastating block by block while trivial citywide.”

Judges perceived the clearest edge in **rhetorical effectiveness**: 8.17/10, +0.51 over the current field.

One explicit adverse case found that the model “never squarely neutralized” an equal-political-difficulty concession.

Gemini 3.8 Flash

A polished, disciplined comparative debater whose recurring weighing framework and reliable formatting make arguments easy to follow. Judges nevertheless perceive a consistent substantive deficit—especially in rebuttal specificity—and its frequent engagement does not always translate into fully answering the opponent’s exact distinction.

Tencent Hy4 Preview

Overall, this is a disciplined, rhetorically effective rebuttal specialist with strong grounding, active weighing, and flexible advocacy, tempered by occasional blunt questioning and unresolved side-swap flags.

"GPT-6 Astra scored 95% on a robot control task, up from Fable 5.1's 40%, with 6.2x fewer output tokens at 2.3x lower cost. On harder tasks involving precision, Astra has the same success rate as Fable 5.1 (2/20), but uses 3.9x fewer output tokens and is 1.6x cheaper. If trends hold, LLMs could control robot arms in real time as soon as the end of this year, or by 2029 at the latest. We urge the research community to look into the implications if general-purpose robots arrive within the next two years. SOTA LLMs are showing surprising generalizability to robotics and their latency is improving rapidly."

- Jay Chooi, Benchmarking robots at RobotCurve

Imagine a future model or system that isn't just a digital AI controlling a computer, it can also input/output practically anything a human can because its cognitive ability matches humans.

So it can control robotics, drones, cars, fighter jets, etc. My definition of AGI is similar to what we had in cinema / science fiction for decades. Jarvis / Friday for example not only ran the entire stark industries, it also could run scientific experiments and simulations, it also ran and controlled tony's entire house, while also being able to control his computer to do computer tasks and finally ability to interface with his robotic suits to control them.

ASTRA is showing tiny early signs of some of that.

But for something to be AGI. I should be able to tell a future version of astra / gpt-6 to:

1. Help me build this IKEA furniture (using live video).

2. Connect my drone and tell it to have the drone follow me.

3. Ability to drive a car to a certain performance metric, doesn't mean it has to drive ~25k without a minor accident which is actually the rate for humans non police reported accidents, or 500k with police reported accident or beat humans million miles between fatality. It just has to be able to drive well. So the benchmarks can be 50 or 100 hours between disengagement.

4. Take-off, Fly & land a plane / jet in a simulator

Astra has clearly made huge progress on the Computer Use and spatial reason. But alot needs to be done to get the latency to real time on all input/output and getting spatial reasoning to match human's visuospatial.

The definition of AGI has always been in times past a AI system that can match or exceed human cognitive abilities across any intellectual task or domain. Notice its not matching human performance nor exceeding it. For example there is only 1 human on earth that can shoot as good as steph curry out of 7 billion.

Its the human cognitive ability matching.

Once these two things (human spatial reason & realtime io to anything) happen.

We Officially Have AGI.

benchmark explanation: https://x.com/adonis_singh/status/2031516746570469837

Notes

• Average Inference Time: 40m 12s
  • Fable 5 averaged 18m 04s
• Total Cost (for 15 builds): $147.55
  • Fable 5 cost $54.93
• Average JSON Size: 34.07 MiB (largest 88.76 MiB)
  • Roughly comparable to Fable's 5 average of 30.65 MiB

Despite no change in API pricing, Fable 5.1 was nearly 3x as expensive as Fable 5 on MineBench. With roughly 2x the inference time, much of that difference appears to come from substantially longer reasoning.

The price increase is quite significant considering Anthropic advertises the same API prices, though it still is massively cheaper than GPT 5.6 Sol P (the current top model on the leaderboards). I find that quite interesting as in my personal usage, GPT 5.6 Sol is extremely efficient with my 20x subscription, though MineBench benchmarked 5.6 Sol P and not the standard Sol variant ^^

There are some builds/styles I (personally) liked better from Fable 5. To me some of Fable 5.1's builds, like the Astronaut, are much closer to Opus 5's style which makes me curious about what it's like coding with Fable 5.1; I'd be very disappointed if Fable 5.1 adopted the Opus 5 style of gibberish english 😭

Also, it was really interesting to see how Fable 5.1 actually was the first model to create genuinely recognizable interiors! Here's a video showing the interior of Fable 5.1's cottage build (you can see a bed, table, bookshelf, and fireplace) – you can explore any build now on MineBench by clicking the joystick icon in the voxelBox header :)

Full release-notes/thoughts on the GitHub release

• If you enjoy these posts please feel free to help fund the benchmark
  • All funds are currently going directly towards API costs for benchmarking new prompts
  • Sharing the benchmark and starring the Git repository also helps :)
  • Alternatively, if you have the API credits, please feel free to add prompts and generations to the gallery and post them around!
    • This is actually preferable to donations to me directly, the hosting expenses and whatnot I've always been able to cover out-of-pocket, just the API costs were hard to cover 😓

Benchmark: https://minebench.ai/
Git Repository: https://github.com/Ammaar-Alam/minebench

Previous Posts:

• Comparison of Map Prompt
• Comparing Fable 5 and Opus 5
• Comparing GPT-5.5 Pro and GPT-5.6 Sol
• Comparing Opus 4.8 and Fable 5
• Comparing Opus 4.7 and Opus 4.8
• Comparing GPT 5.4 and GPT 5.5
• Comparing Kimi K2.5 and Kimi K2.6
• Comparing Opus 4.6 and Opus 4.7
• Comparing GPT 5.4 and GPT 5.4-Pro
• Comparing GPT 5.2 and GPT 5.4
• Comparing GPT 5.2 and GPT 5.3-Codex
• Comparing Opus 4.5 and 4.6, also answered some questions about the benchmark
• Comparing Opus 4.6 and GPT-5.2 Pro
• Comparing Gemini 3.0 and Gemini 3.1

Extra Information (if you're confused):

Essentially it's a benchmark that tests how well a model can create a 3D Minecraft-like structure.

So the models are given a palette of blocks (think of them like legos) and a prompt of what to build, so like the first prompt you see in the post was a fighter jet. Then the models had to build a fighter jet by returning a JSON in which they gave the coordinate of each block/lego (x, y, z). It's interesting to see which model is able to create a better 3D representation of the given prompt.

The smarter models tend to design much more detailed and intricate builds. The repository readme might help give a better understanding.

(Disclaimer: This is a public benchmark I created, so technically self-promotion :)

Frontier AI seems broadly overconfident about its own ability. This benchmark helps show by how much.