u/COAGULOPATH

It's fast and the benchmarks look good, mostly surpassing Gemini 3.1 Pro. But it's very expensive for a Flash model.

>It achieves speeds of over 280 output tokens/s, but higher token usage and token pricing make it over 5x more costly to run the Intelligence Index than Gemini 3 Flash, and 75% more costly than Gemini 3.1 Pro. Gemini 3.5 Flash is $1.50/1M input and $9/1M output tokens, Gemini 3 Flash was $0.5/$3 per 1M input/output tokens, a 3x increase. The rest of the increase was driven by higher token usage when running our benchmarks

This model is apparently now powering AI Mode in Search, which surprises me: I wouldn't have thought that would be affordable.

Also:

>We’re also hard at work on 3.5 Pro. It's already being used internally, and we look forward to rolling it out next month.

Both models spent $10,000 (the limit). GPT-5.5 scored 0.4% and Opus 4.7 scored 0.2%.

This benchmark is quite difficult for clankers. It seems almost pointless to test current LLMs on it: they all score equally (about zero). My prediction of a 30% score in a year seems unlikely to come true.

It's probable that new breakthroughs (or at least much better base models) are needed here. (That said, when LLMs finally do chip a dent in ARC-AGI-3, even a little one, expect scores to shoot to 100% quite fast)

So far, so boring.

Less boring is the ARC Prize's analysis of how GPT-5.5 and Opus 4.7 played, based on reasoning from 160 games. The two models failed in extremely unlike ways.

Opus 4.7 aggressively theorycrafts, and learns game mechanics fairly well. But it assumes facts not in evidence, struggles to integrate new data into existing beliefs, and often can't (or won't) backtrack out of wrong assumptions. It ends up playing from a theory of the game that is "neat, plausible and wrong."

GPT-5.5 just...doesn't commit to a theory. Ever. It taps buttons but never seems to learn anything. In every turn, it sounds like an old man who has woken from a deep slumber and is seeing the game for the first time ("I'm analyzing a game with a grid..."). It blindly wonders if it's playing Tetris, or if the orange blocks are lava. Everything gets pattern-matched onto some existing videogame, with its previous reasoning forgotten.

It's funny that GPT-5.5 "doubles" Opus 4.7's score. To the extent this isn't noise, it's likely due to GPT-5.5's exploration-focused approach getting luckier a little more often.

tldr: Opus 4.7 is precise but inaccurate, GPT-5.5 accurate but imprecise.

Do tests like ARC-AGI-3 mean much, in the end? I'm not sure. I suspect the games were designed (in part) to focus around things that humans find easy and LLMs find hard, like spatial reasoning. But many important things (like robotics) involve spatial reasoning: I see this as defensible.

(I got around 80% on the two games I played. According to its creator, "Any smart human giving it real effort should score >90% on ARC-AGI-3". y u bully me man :( )