u/Correct_Tomato1871

Added 3 new runs to the MindTrial leaderboard:

• GPT-5.6 Sol: 85/98, 0 errors, 1h09m
• GPT-5.6 Sol Pro: 87/98, 0 errors, 1h54m
• Grok 4.5: 72/98, with 3 errors, in 4h17m

GPT-5.6 Pro achieved the highest score by a single model on the leaderboard. Its 87/98 matches the overall high set by OpenRouter Fusion, which uses a panel of models plus a judge rather than a single model working alone.

Both GPT-5.6 configurations went 39/39 on text. Pro mode improved visual performance from 46/59 to 48/59.

The Pro trade-off was interesting: two extra passes, but 64% more runtime and nearly 3× the recorded output tokens. It actually used fewer Python calls than standard mode, so the additional work appears to have happened mostly inside the model.

It was not uniformly better. Only four task outcomes changed: three gains and one regression. On the regression, Pro spent more than 12 minutes and all 10 tool calls but still turned a standard-mode visual pass into a failure.

Grok 4.5 is a clear upgrade over Grok 4.3 and 4.20, which both scored 64/98. It reached 39/39 on text and improved visual performance to 33/59. The weak point was tool-loop control: 583 Python calls, 47 tasks at the 10-call cap, and every one of its 26 non-passes occurred on a cap-hitting task.

My takeaway: standard GPT-5.6 has the best speed/quality balance—one pass behind GPT-5.5, but roughly twice as fast and using about half the combined tokens. Pro mode sets the top single-model score, but the two-pass gain is expensive; Grok 4.5 improved substantially but still struggles with visual/spatial tasks and stopping its tool loops.

Added Anthropic Claude Sonnet 5 to my MindTrial leaderboard.

Result: strong text performance, but a more uneven full benchmark profile.

• Overall: 72/98
• Text: 38/39
• Original visual set: 22/33
• New visual2 set: 12/26
• Hard errors: 6
• Runtime: 5h 31m

The good: text reasoning is excellent. 38/39 puts Sonnet 5 in the top text tier in this run.

The weaker part: visual2. It scored 12/26 there, compared with 16/26 for Gemini 3.5 Flash, 18/26 for Claude 4.8 Opus, and 22/26 for GPT-5.5 / Gemini 3.1 Pro.

Gemini 3.5 Flash is probably the more interesting practical comparison here than Opus/Fable. Flash scored 77/98 overall, matched Sonnet 5 on the original visual set at 22/33, did better on visual2, had fewer hard errors, and finished much faster: 2h 13m vs 5h 31m.

The surprising part was efficiency. Sonnet 5 used Python heavily, but on hard visual tasks it often hit the 10-call tool cap and then ran into max-token / no-actionable-content errors.

Main takeaway: Claude Sonnet 5 looks very strong on text, but in this benchmark it was less efficient and less robust on visual2 than I expected from a Sonnet-class model.

I tested OpenRouter Fusion on MindTrial - OpenRouter’s multi-model deliberation feature where an outer model can ask a GPT/Claude/Gemini-style panel for help, then use a judge model to synthesize the result.

I ran two Fusion configurations:

• Default-reasoning Fusion: gpt-latest outer, general-high panel, Claude Opus judge. Result: 87/98, with 10 fails and 1 hard error. Runtime: ~3h54m.
• High/xhigh Fusion: GPT-5.5 high outer, Claude Opus/GPT/Gemini Pro xhigh panel, Claude Opus xhigh judge. Result: 86/98, with 12 fails and 0 hard errors. Runtime: ~10h18m.

For comparison, standalone GPT-5.5 high scored 86/98, with 7 fails and 5 hard errors. Runtime: ~2h18m.

Main finding: Fusion helped reliability, but not accuracy. It reduced hard errors, but mostly converted them into ordinary wrong answers rather than extra passes.

The “panel union” hypothesis also did not hold. The high/xhigh Fusion run still missed 7 tasks that at least one standalone panel comparator solved.

The likely bottleneck is invocation policy: Fusion was optional, not forced. Based on the log, it appears to have been called on only a small minority of tasks, often late after many Python attempts and a lot of accumulated context.

Main takeaway: OpenRouter Fusion looks promising as a reliability layer, but in this benchmark it was not an oracle over its panel - and xhigh deliberation made the long-tail visual/spatial tasks much more expensive without improving the aggregate score.

Added Z.AI GLM 5.2 to my MindTrial leaderboard.

Result on the 39 text-only tasks:

• GLM 4.7: 13/39
• GLM 5: 27/39
• GLM 5.1: 32/39
• GLM 5.2: 30/39

So GLM 5.2 does not beat GLM 5.1 on strict raw score. It finished with 30 passes, 7 failures, and 2 hard errors.

But the speed difference is the interesting part:

• GLM 5.1: about 4h04m
• GLM 5.2: about 40m46s

That is roughly a 6x speedup, with only a 2-task drop in raw score.

Manual review also suggests several GLM 5.2 failures were not pure reasoning misses, but rather "output-discipline" issues. In multiple cases, the right answer seemed to be present in the output, but not in the exact required final format.

Main takeaway: GLM 5.2 looks like a major practical speed upgrade over GLM 5.1, but not a clean strict-benchmark upgrade on MindTrial text reasoning.

Added Anthropic Claude Fable 5 to my MindTrial leaderboard.

This is a strong Anthropic update:

• Claude Fable 5: 80/98 overall, 0 hard errors
• Claude 4.8 Opus: 73/98 overall, 5 hard errors
• Text tasks: Fable hit 39/39, vs 35/39 for Opus 4.8
• Runtime improved a lot too: ~3.02h for Fable vs ~5.03h for Opus 4.8

It lands right in the top tier of the 98-task board:

• GPT-5.5: 86/98
• Gemini 3.1 Pro: 81/98
• Claude Fable 5: 80/98
• GPT-5.4: 80/98
• Gemini 3.5 Flash: 77/98
• Claude 4.8 Opus: 73/98

The interesting caveat: Fable did not clearly improve the newer visual2 subset. It scored 17/26 there, slightly below Opus 4.8 at 18/26 and well below GPT-5.5 / Gemini 3.1 Pro at 22/26.

Tool use looked cleaner overall: fewer Python calls than Opus 4.8 and fewer 10-call cap hits. So the main gain seems to be reliability, speed, text performance, and original visual tasks — not a clean sweep on the hardest new visual2 tasks.

Main takeaway: Claude Fable 5 is a real Anthropic leap in MindTrial, but not the new overall leader.

Added 3 new models to my MindTrial leaderboard:

• ByteDance Seed 2.0 Lite: 67/98 overall, with 32/39 text, 16/33 original visual, and 19/26 visual2. Big jump from Seed 1.6’s 45/98.
• MiniMax M3: 30/39 text-only. Clear improvement over MiniMax M2.7’s 23/39.
• StepFun Step 3.7 Flash: 33/98 overall. Fast, but weak across text and vision.

Main interesting finding: Seed 2.0 Lite did especially well on the newer visual2 tasks, but took almost 15 hours and used Python heavily.

Most surprising finding: StepFun had Python tool access but made zero Python calls, despite being positioned as an agentic/tool-capable model.

Main takeaway: Seed 2.0 Lite is the real addition here; MiniMax M3 deserves a proper visual rerun; StepFun 3.7 Flash needs another look at configuration/tool behavior before drawing strong conclusions.

Added Anthropic Claude 4.8 Opus to my MindTrial leaderboard, run with xhigh adaptive thinking and Python tool use.

Result: 73/98 overall

• Text: 35/39
• Original visual/subjective-visual: 20/33
• visual2: 18/26
• Hard errors: 5
• Runtime: ~5h02m

Compared with previous Opus runs:

• Claude 4.6: 69/98, 12 errors
• Claude 4.7: 69/98, 9 errors
• Claude 4.8: 73/98, 5 errors

So 4.8 is the best Claude Opus result so far on this expanded 98-task board. The improvement mostly comes from fewer hard errors and better visual performance, not a big jump in text reasoning.

The surprising comparison is Gemini 3.5 Flash:

• Gemini 3.5 Flash: 77/98, 1 error, ~2h13m
• Claude 4.8 Opus: 73/98, 5 errors, ~5h02m

Claude 4.8 wrote cleaner Python and had far fewer code/runtime errors, but Flash was much faster and more aggressive with tool use — and still scored higher overall.

Main takeaway: Claude 4.8 is a cleaner, stronger Opus run, but not a MindTrial breakthrough.

Added 3 new models to my MindTrial leaderboard:

• Gemini 3.5 Flash: 77/98 overall, with a perfect 39/39 on text. On the original 72-task suite it actually beats Gemini 3.1 Pro, 61/72 vs 59/72, and is much faster (~1h17m vs ~2h44m). But visual2 (set of new visual tasks in grayscale with higher resolution) flips the story: 3.1 Pro scores 22/26 there, while Flash gets 16/26. Full 98-task board: 3.1 Pro 81/98, Flash 77/98.
• Qwen3.7-Max (text-only): 32/39 text. Big step up from Qwen3.6 Plus Preview (26/39) and Qwen3-Max (20/39), but with 5 hard errors from structured-output / JSON issues.
• Qwen3.5-Omni-Plus: 54/98 overall. Not frontier-level, but 15/26 on visual2 was surprisingly solid — only one pass behind Flash on that slice.

Main takeaway: Gemini 3.5 Flash is the interesting one. It is fast and very strong, but the newer visual2 tasks still separate it from Gemini 3.1 Pro.
Flash seems to win partly by moving fast — it used Python heavily, often hit the 10-call cap on visual tasks, and still finished faster than Gemini 3.1 Pro, though the code path looked more like “iterate quickly until it works” than “write one robust script.”

I added 26 new visual tasks to MindTrial, under the visual2 prefix.

These are grayscale, somewhat higher-resolution image tasks covering OCR, spatial reasoning, numerical awareness, visual deduction, and pattern completion. All tested models had access to the same Python tool environment.

Because the merged leaderboard now includes models with different task counts, I’m focusing on percentages rather than raw totals.

Old visual → New visual2 pass rate:

• GPT-5.5: 78.8% → 84.6% (+5.8 pts), runtime/task +50.9%
• Gemini 3.1 Pro: 63.6% → 84.6% (+21.0 pts), runtime/task -38.3%, 0 hard errors
• GPT-5.4: 66.7% → 73.1% (+6.4 pts), runtime/task +6.8%
• Claude 4.7 Opus: 51.5% → 65.4% (+13.9 pts), runtime/task -21.3%
• Kimi K2.6: 39.4% → 61.5% (+22.1 pts), runtime/task -13.8%
• Grok 4.20 Beta: 36.4% → 57.7% (+21.3 pts), runtime/task +178.1%

Main takeaway: GPT-5.5 and Gemini 3.1 Pro are basically co-leaders on this new visual slice.

GPT-5.5 had the better accuracy on completed tasks: 88.0% vs. Gemini’s 84.6%.

Gemini had the cleaner reliability profile: same 84.6% pass rate, 0 hard errors, and much better runtime compared with its old visual-task run.

Kimi K2.6 is also interesting: big improvement and strong completed-task accuracy, but still hurt by hard errors and long runtime.

Overall, visual2 seems to be doing what I hoped: OCR is now mostly solvable for top models, while spatial reasoning and visual pattern completion still separate the field.

Selected models on visual2tasks: http://www.petmal.net/shared/mindtrial/results/2026-04-28/mindtrial-eval-selected-models-visual2-tasks-04-2026.html

Added 2 major models to my MindTrial leaderboard: OpenAI GPT-5.5 and DeepSeek V4 Pro.

GPT-5.5 takes the top full-benchmark spot in this run:

• Overall: 64/72 passed, 88.9% pass rate, 94.1% accuracy
• Text-only: 38/39
• Visual: 26/33
• Runtime: 1h 9m total, ~20.1s median per task

Compared with GPT-5.4, that is +3 overall passes, +4 visual passes, fewer hard errors, and a big speed jump: 3h 10m → 1h 9m.

It also used fewer Python calls: 247 → 133, with much lower median input/output tokens than GPT-5.4. So this looks less like brute-force tool exploration and more like more restrained/efficient tool use.

One caveat: GPT-5.5 was run at high reasoning, not xhigh, following OpenAI’s GPT-5.5 guidance for hard reasoning tasks. It also had 4 hard errors, all invalid_prompt usage-policy flags on visual tasks — likely false positives, but still real benchmark reliability misses.

DeepSeek V4 Pro also looks like a major text-only upgrade:

• Text-only: 37/39
• Visual: skipped
• Hard errors: 0
• Runtime: 2h 14m

Compared with DeepSeek-V3.2, it went from 32/39 to 37/39 on text tasks and eliminated 6 hard errors.

Main takeaway: GPT-5.5 is the new full MindTrial leader here — and notably fast for that score. DeepSeek V4 Pro is a strong and much cleaner text-only DeepSeek run, but not comparable as a full multimodal entrant in this setup.

Added 3 new models to my MindTrial leaderboard:

• Claude 4.7 Opus: 52/72 overall. Strongest of the new additions, but still behind GPT-5.4, GPT-5.2, Gemini 3.1 Pro, and Claude 4.6 in the current board.

• Kimi K2.6: 50/72 overall, with 37/39 text and 13/33 visual @ 32k max-token cap. Better than the included K2.5 run at 42/72, but that K2.5 run used a 16k max-token cap. In an internal K2.5@32k rerun, K2.5 reached 47/72, so the gap shrank from 8 passes to 3. K2.6 also took over 9.5 hours, which is a big part of the story.

• Xiaomi MiMo-V2.5: 31/72 overall, with 21/39 text and 10/33 visual. Better than MiMo-V2-Omni (29/72), mostly thanks to vision, but still nowhere near the top multimodal models.

Main takeaway: useful leaderboard movement, but more evolution than revolution this round.