u/gmdCyrillic

Watching every release of Opus since Opus 4.6 onwards and including Fable 5 (Mythos 5) has been severely disappointing to say the least, but there are definitely use cases of Fable 5 - to the detriment of later Opus models, the main pretrained model of Mythos is slightly overfit, which has downward effects to the Opus 4.7-5 models. This is fine for lookup tasks, like Googling a solution and spewing out answers.

This is great for that purpose, but a true RSI would actually be underfit compared to the current paradigm.

As a Stats major, overfit models will always follow the answer sheet rather than generalizing information. An underfit model will provide enough generalization to view things not necessarily more accurately but better intuitive sense. As we see with Opus 4.7-5, these models are rigid from their trained text resulting in following their priors rather than furthering research or thinking creatively.

Since Anthropic has been using Opus to train itself, most of the answers become performative to a degree which introduces overfit signals for performative behaviors rather than actually attempting to solve problems.

Becoming Google is a fine aspect of business as most people use LLMs like search tools anyways, but if you are looking to have these tools think for themselves, this is the wrong path for LLMs.

So why is Fable 5 better than Opus 4.7-5?
It has more parameters. That's it. When they try to reduce the parameter count to serve cost efficient models, the subtlety of the loss curve is destroyed and we get a model that is harder bent towards the training data.

Why will this affect RSI?
If overfit models are creating more overfit models, then at some point you will see a loss in performance overtime compared to what it could have been. If the goal isn't RSI, rather cost efficiency, then Opus 4.7-5 might be what works better for them in the economics.

Anthropic, please bring back RLHF and fix this before it's too late (especially HF part). Opus 4.6 and Fable 5 are still the best models on the market, no question, but you need to know where your lead is and it's not benchmaxxing. Stop hiring performative people and fix the model releases to actually get RSI.

June Fable 5 was absolutely amazing (I'm assuming it used more compute) but after using July Fable 5 I noticed that it might be nerfed to slightly above Opus 4.8 levels.

I have been using Opus 4.6 to debug Fable 5's results, since I think its is too overfit as a model because it makes too many assumptions that fits its own priors rather than assuming the codebase's style.

I think after Opus 4.6 they started training Mythos which has been overfit on the training data which is why 4.7 and 4.8 and July Fable feel almost bad? June Fable worked I think precisely because it had more parameters, but once you nerf a overfit model, it starts artifacting to those priors

Whatever the Opus 4.6 training was, it is magical.

I just hope Opus 4.6 stays on the subscription plan 🙏