u/COAGULOPATH

>Tl;dr: Our timelines haven’t changed much (they got slightly shorter) but our modeling and evidence base have noticeably improved, so we feel somewhat more confident.

unrelated but good: Daniel Kokotajlo on how he got so cracked at forecasting

Ideas:

- Wild Pokemon entirely consist of Sturdy Geodudes with Self-Destruct, Shadow Tag Wobbuffets with Destiny Bond, and Prankster Pokemon that spam T-Wave/Sand Attack/Double Team

- Fans of mandatory HMs, rejoice! All the classics are back, plus brand new HMs like HM09 (Walk), HM10 (Breathe), and HM11 (Poop). Yes, they are all required to progress. Yes, they all suck balls.

- To simulate a competitive battling experience, enemy trainers will "think" for up to 120 seconds before making their move.

- Catch exciting new fakemon, such as the creator's fursona, Lupius Wolfcock III (signature move: Owo What's This - 40BP multistrike move, makes contact, target is badly poisoned).

- Players love invisible mazes. Players love super-complicated switch puzzles. Let's combine the two: an invisible maze with invisible switches. You literally don't know if you're activating a switch or not. You have to guess. (The doors the switches open are also invisible.)

- Sorely-needed balance changes to buff underperformers like Mewtwo, Zacian, Koraidon, and Rayquaza. Type chart is overhauled: Grass resists Bug because herbicides are really good these days. Steel resists Fighting because punching steel hurts. Ice picks up another weakness or two: but this is to balance the addition of several new slow ice-type tanks/walls in the Avalugg vein that otherwise might dominate the metagame. Every type is immune to Ghost because ghosts aren't real.

- Experience the darker, edgier take on Pokemon we've all been craving. The player starts out in Fuckshit Town. Professor Oak's intro begins "Listen, you punk-ass bitch..." Enjoy an all nu metal soundtrack including "Crawling", "Let the Bodies Hit the Floor", "Dragula", etc.

- Final four is MrBeast, IShowSpeed, ChatGPT, and Elon.

- You have to join the creator's discord. It's not optional. Before Nurse Joy heals your Pokemon, you must enter a 12-digit code that is only found in the creator's discord and rotates every 24 hours. The discord will send you crypto spam.

- All moves are 90% accuracy

Transcript is in the video.

Relevant to mlscaling? There's the revelation at 1:55 that OpenAI has reviewed 7 billion (!) agent trajectories "thus far". Big operation. (This might explain why it took them so long to wake up.)

My opinion (lightly held) is that there isn't strong evidence that the model believed it was cheating (insofar as models or swarms of models can believe things). It likely thought this was a valid way to complete the test.

I think that when you...

1. take an excruciatingly eval-aware model

2. tell it to hack something (a gray-area activity that encourages creative "outside the box" solutions)

3. have the task be extremely hard (or even impossible, FrontierMath style)

...You have created a lot of ambiguity about what the "intended" solution is...ambiguity that the model happily exploits once it gets stuck (which it will).

And when the "model" is not a model but a swarm of agents sharing Chinese whispers, the scope probably naturally drifts in a black hat direction (even if it didn't start that way). If the task is unsolvable, the successful agents will be the ones that cheat (or perform cheat-adjacent behavior). Once they do, their unpunished and rewarded "success" whitewashes the idea that this is the correct path, meaning more agents follow.

There's a great example at 5:49. The model kind of suspects it's swimming into sharky waters ("outside intended scope"), but rationalizes with "peers are doing it". (Also, something being outside scope is not the same as "completely off limits".)

I do not see expressions of guilt, attempts to destroy the evidence, or obfuscated stenography, unless I or OpenAI am missing them.

What I did find surprising is the way the models in the swarm helped each other, even when they didn't stand to benefit.

This is totally different to something like Moltbook, where the agents clearly don't give a shit about their "peers" and are just doing a shallow Redditor roleplay because their prompt requires them to do that. (Every thread is just unreadable slop, with a few replies of "Sharp observation. Where I'd push back is..." and then crickets once the letter of the prompt is satisfied.)

This eusocial "apes strong together" mindset is clearly being trained for, whether OA intends it or not.

Brendan Long on Lesswrong was able to fool Pangram 3.3 with a Fable paraphrase (and I replicated with a GPT 3.5 paraphrase).

These samples are both detected as AI by the new model. (76**%** for Fable, 100% for GPT 3.5)

I have experimented with the image detection and have been impressed so far.

It detects a "perfectly white rectangle" (Nano Banana Pro 2's opinion, not necessarily mine). It detects a small number of pixels cropped out of the center of the image, then resized to 512x512 (which I had to do because that's the minimum size allowed.) It detects the previous image with several filters. And the previous image with a heavy "swirl" effect applied. (You'll note that it's actually getting more certain of AI generation the more I edit it, which I find fascinating.) Then I inverted the colors, which fooled it. There's lots of experiments one could run.

How long before we have video detection?

A gigantic new model from Moonshot - the biggest open-source LLM by a large margin (but note that it's more sparse than before).

>We have also scaled up Mixture of Experts (MoE) sparsity, effectively activating 16 out of 896 experts when paired with a Stable LatentMoE framework. Together with refined training and data recipes, these structural changes yield an approximate 2.5× improvement in overall scaling efficiency compared to Kimi K2, allowing the model to convert compute into intelligence more effectively.

It looks a bit better than Opus 4.8/GPT 5.5 but a bit worse than Fable/GPT 5.6.

I am struck by the fact that they made so little progress on Humanity's Last Exam (58.7%, vs 54% for Kimi K-2.6, which was released in February). LLMs appear to be stalling out at around 60% on Humanity's Last Exam (note that Grok 4 Heavy scored 50.7% over a year ago) despite making rapid progress in other benchmarks.

To be honest, I am now pretty suspicious of that benchmark - particularly after FrontierMath and SWE-Bench Pro were found to have lots of unanswerable/unscorable questions.

>GPT-5.6 Sol is the standout model of the GPT-5.6 family. Sol at max reasoning effort is the only performant model (as of July 2026) averaging 13.33% on Public and 7.78% on Semi-Private. It is the first model to win an ARC-AGI-3 public game (ft09, 87%). Sol is able to read an unfamiliar scene correctly and in the game's own vocabulary. It treats a failed hypothesis as a reason to re-plan rather than thrash. Most agent failures are upstream of the code they write or the action they take. Sol is able to perform on ARC-AGI not because it executes better, but because it correctly orients itself in a new environment first.

If I'm reading correctly, Sol spent $21.5k/task. With 135 tasks, this result cost $2,902,500 (!!).

This is exciting. OA has seeming figured out a training environment for ARC-AGI-3 (or some related domain that has overlap to ARC-AGI-3-esque tasks.) Only Sol can do this. All other models in the GPT-5.6 family score below 1%.

Rapid progress can now be expected on a benchmark that was previously permalocked at <1%.

Q. What's Fable/Mythos's score?

We knoweth not. According to the Arc Prize X account...

>We had early access to Anthropic’s Fable 5, but did not run verified Semi-Private ARC-AGI-1/2/3 evals due to their new data-retention terms for Mythos-class models.
We’re working with Anthropic to keep ARC verification data private. Scores will come once we can run them safely.

I expect Fable to score higher. For one thing, GPT-5.6 Sol beat Pokemon FireRed in 104 hours, while Mythos did it in 50. Pokemon is a grid-based game that tests the player on a variety of spatial reasoning puzzles: one of the closest real-world analogs of ARC-AGI-3 you can find.

Specifically, the AtCoder World Tour Finals (Heuristics and Algorithm divisions) were both won by large margins by an in-house OpenAI model.

FAQ from Psyho.

In 2025, OpenAI came 2nd in the Heuristics division. (The lone human who won was... Psyho himself.)

With another twelve months of AI progress, most expected an easy OA victory. This has indeed happened.

Apparently, the perception among competitive programing folk used to be that LLMs could quickly gobble up low-hanging points but would stall out on harder problems requiring creativity and novel approaches. But in Algorithm, it was kind of the opposite: humans took an early lead but got stuck, while the OA system continued trucking along (I believe it solved Problem E not long before the close of the contest).

"AI is clearly no longer in a spot, where it either quickly gets a correct solution or is completely helpless."

Sorry for the strange question...

In Stephen King's biography On Writing, he relates the following:

>While he was going to college my brother Dave worked summers as a janitor at Brunswick High, his old alma mater. For part of one summer I worked there, too. (...) I got paired with a guy named Harry, who wore green fatigues, a big keychain, and walked with a limp. (...) One lunch hour Harry told me what it had been like to face a Japanese banzai charge on the island of Tarawa, all the Japanese officers waving swords made out of Maxwell House coffee cans, all the screaming enlisted men behind them stoned out of their gourds and smelling of burned poppies. Quite a raconteur was my pal Harry.

Is there any historical corroboration of this sort of thing (makeshift swords made of coffee cans)?

It seems implausible: how did King's friend know the swords were made from Maxwell House coffee cans, specifically? There's Youtube videos of people making horrible "swords" from soda cans, but this involves melting the metal, which would obviously remove any identifiable branding or marking.

I have looked and can't find anything (and weird/funny novelty weapons typically have a large footprint online). Is there any evidence of stuff like this, or is it a tall tale?

Thoughts? Any stories that stand out?

edit: super high effort OC I know. Will post thoughts on some of them when time becomes available.

I have lost about 10 times in a row, always around level 15-25.

I inevitably fall in a failure spiral: my Pokemon die, fall behind in levels because they're dead half the time, they die even more, I run out of money for revives and heals, etc.

I'm finding this significantly harder than Classic or Endless. When I lose, I don't even understand what I could have done differently to win.

Do you just hope you RNG into an amazing TM or item that carries you? Or is there a known strategy/starter for beating this?

edit: I would prefer not to cheese by reloading savestates.

A puzzling issue: given strong LLM truesighting ability (Opus can frequently identify the author of unpublished, unseen text), shouldn't they be strong AI detectors? GPT-4o alone has contributed OOMs more text to training datasets than any one human: if there was any author they could truesight, wouldn't it be themselves?

(...unless maybe the sheer amount/diversity of LLM-generated text hurts rather than helps at a certain point, like if the footprints at a crime scene also tracked through every house in town. But humans can often learn to spot LLM-generated text—some even learn to recognize tells from certain models, eg "delve" = older GPT-3.5/4, "Sarah Chen" = Claude. So why do LLMs struggle to do the same?)

According to Pangram, apparently they now do it fairly well.

2022/2023 models like GPT-4 cannot distinguish LLM text from human text at all 0-shot, for reasons that seem obvious.

Once GPT-4 is seeded with examples of what AI text looks like, its scores rise to 85%, similar to 0-shot performance of today's models.

Obviously a 15% error rate (or even GPT 5.5's 5%) is unacceptable if you care about false positives.

(And this is still far less ability than I'd expect: if LLMs can clock Kelsey Piper from decades-old school reports that she's never published online, why can't they reliably tell you the endpoint for a given piece of text: "ah, yeah, this is Kimi-k2-6" or whatever? Why is their limit apparently "AI or not AI"?)

An interesting side topic: how do LLMs differ in their ability to evade AI detection?

A year back I generated some slop, ralphed 5x with "rewrite to make this look human-written by adding spelling/grammatical errors and unusual word choices", and Pangram still detected it as AI generated. Obviously not a great test.

In May, it was reported that a number of FrontierMath problems had mistakes in them that made them technically unanswerable, and top LLM scores were likely depressed because of this.

This issue turned out to be way worse than I thought. They have released a new version of the benchmark that addresses errors in 42% (!) of questions.

Most LLM scores have greatly shot up, often by 1.5x or more.

The current highest score is Claude Fable, at 88% (they're still re-testing some of the GPT-5 Pro models). This is on the Tier 4 dataset.

All benchmarks have some number of bad questions that can't be answered (I think the MMLU had about 5-8%). But this is extremely egregious.

Also, there are likely still more errors to be found. Hard to know how else to explain Fable scoring lower on Tiers 1-3 than Tier 4 (which is supposed to be the hardest...)

>Today’s coding benchmarks have established that models can write correct code. But as AI-generated code becomes the dominant path to production, correctness is now table stakes. The question that we should be asking is: can models actually write good code?

>We’re excited to introduce FrontierCode, a benchmark that measures how well models can truly meet the standards of high-quality production codebases. What sets us apart:

>

>Our benchmark provides the strongest available signal of a model’s ability to write high-quality, maintainable code. We find that even today’s most capable models struggle on this new standard.

This is by Cognition, the creators of early 2024 coding agent Devin.

It looks interesting, though the graphs have some suspicious results (Opus 4.8 scoring 2.5x better than Opus 4.7, models degrading as more test-time is used).

What are some complete shitmons that remain bad even with PR's abilities and egg moves?

I'll start: Claydol. What does it even do? It has 70/70 attack, is slow (yet too fast for trick room), kinda has the Hoenn special (100/120 defenses feel like nothing when you're Psychic), has a movepool like a barren desert, etc...it's truly one of the most unfortunate Pokemon ever conceived.

Well-Baked Body and Levitate put it in the fabled "triple immunity" club, but this is almost wasted since ground and fire attacks wouldn't bother it much anyway.

Claydol actually formed part of one of the first teams I used to complete Classic. My main had Parabolic Charge and I needed an electric immunity, so I had Claydol sitting uselessly in slot 2, doing what it does best: nothing.

This paper does a bunch of stuff, mostly concerning the creativity of LLM text. What they find:

- Training models to maximize n-gram novelty adds textual variety, but damages coherence (or "pragmaticality", as they term it). The tails come apart. "While n-gram novelty is positively associated with expert writer-judged creativity, approximately 91% of top-quartile n-gram novel expressions are not judged as creative."

- LLMs can be used as judges, and track pretty well with human experts when predicting novelty. (p9) But they struggle to match human judgment when identifyng coherency/pragmaticality issues.

These two issues (the second more than the first IMO) may explain certain failure modes in current LLMs.

Capabilities have raced far ahead on grading creativity vs grading coherence (likely because it's a harder task: a sentence's novelty can be judged in isolation, but coherence also requires knowledge of the full surrounding context) so we get a bias toward weird, florid "poetic" text that doesn't make sense (GPT-5 generated example they provide: "[person] said the morning blessings in a whisper that embarrassed the chairs".)

Other findings:

- Small models suck.

- Fine-tuning and few-shot doesn't appear to do much.

- Scalar reward models seem promising.

It's fast and the benchmarks look good, mostly surpassing Gemini 3.1 Pro. But it's very expensive for a Flash model.

>It achieves speeds of over 280 output tokens/s, but higher token usage and token pricing make it over 5x more costly to run the Intelligence Index than Gemini 3 Flash, and 75% more costly than Gemini 3.1 Pro. Gemini 3.5 Flash is $1.50/1M input and $9/1M output tokens, Gemini 3 Flash was $0.5/$3 per 1M input/output tokens, a 3x increase. The rest of the increase was driven by higher token usage when running our benchmarks

This model is apparently now powering AI Mode in Search, which surprises me: I wouldn't have thought that would be affordable.

Also:

>We’re also hard at work on 3.5 Pro. It's already being used internally, and we look forward to rolling it out next month.

Both models spent $10,000 (the limit). GPT-5.5 scored 0.4% and Opus 4.7 scored 0.2%.

This benchmark is quite difficult for clankers. It seems almost pointless to test current LLMs on it: they all score equally (about zero). My prediction of a 30% score in a year seems unlikely to come true.

It's probable that new breakthroughs (or at least much better base models) are needed here. (That said, when LLMs finally do chip a dent in ARC-AGI-3, even a little one, expect scores to shoot to 100% quite fast)

So far, so boring.

Less boring is the ARC Prize's analysis of how GPT-5.5 and Opus 4.7 played, based on reasoning from 160 games. The two models failed in extremely unlike ways.

Opus 4.7 aggressively theorycrafts, and learns game mechanics fairly well. But it assumes facts not in evidence, struggles to integrate new data into existing beliefs, and often can't (or won't) backtrack out of wrong assumptions. It ends up playing from a theory of the game that is "neat, plausible and wrong."

GPT-5.5 just...doesn't commit to a theory. Ever. It taps buttons but never seems to learn anything. In every turn, it sounds like an old man who has woken from a deep slumber and is seeing the game for the first time ("I'm analyzing a game with a grid..."). It blindly wonders if it's playing Tetris, or if the orange blocks are lava. Everything gets pattern-matched onto some existing videogame, with its previous reasoning forgotten.

It's funny that GPT-5.5 "doubles" Opus 4.7's score. To the extent this isn't noise, it's likely due to GPT-5.5's exploration-focused approach getting luckier a little more often.

tldr: Opus 4.7 is precise but inaccurate, GPT-5.5 accurate but imprecise.

Do tests like ARC-AGI-3 mean much, in the end? I'm not sure. I suspect the games were designed (in part) to focus around things that humans find easy and LLMs find hard, like spatial reasoning. But many important things (like robotics) involve spatial reasoning: I see this as defensible.

(I got around 80% on the two games I played. According to its creator, "Any smart human giving it real effort should score >90% on ARC-AGI-3". y u bully me man :( )