r/mlscaling

A frozen cross-vendor study of 31,430 trials across 11 GPT, Claude, Gemini & Kimi Large Language Models found 11,658 successful executions with exactly zero visible UTF-8 output bytes.

Across 4,290 strict matched semantic pairs, null-condition arms produced 2,505 Voids; matched output-licensed controls produced 0.

These were not refusals, safety blocks, rate limits, or transport failures.

Raw records, event hashes, verification code, and full analysis are public.

Wanted to know what different models actually cost across the whole market. Numbers turned out really interesting.

The spread.

Cheapest output on the platform is Mistral Nemo, $0.03 per million tokens. Most expensive is o1-pro at $600. I re-ran that twice because it looked like a units bug. Median paid model is about $2, so most of the catalog sits down near the floor and there's a thin little line of stuff way up at the top.

Provider averages, with a caveat.

• OpenAI: $47.63
• Anthropic: $44.79
• Google: $5.58
• Mistral: $3.68
• Qwen: $2.86
• Meta: $0.74

Caveat first because someone will say it anyway: these are averages over each provider's catalog, not weighted by what people actually run. OpenAI's number is dragged way up by o1-pro, which I doubt anyone is using at volume. Blended is 3:1 input to output, which is roughly what my own usage looks like.

Even so, Meta at $0.74 against OpenAI at $47.63 is a 64x gap. For the stuff I use models for (mostly code and summarizing), I don't get 64x anything.

Output tokens are where reasoning models get you.

Input and output are priced separately, and on the thinking models the ratio gets silly. Qwen3's thinking variants are $0.20/1M in and $2.40/1M out, so 12x. Gemini 2.5 Flash is 8.3x. Fine if you're sending one question. Less fine if you've got an agent looping thirty times and every step is paying the output rate. I got a bill like that once and it took me an embarrassingly long time to work out why.

19 free models, and a few are usable.

Not trial-credit free, actually free on the API:

• NVIDIA Nemotron 3 Ultra, 1M context
• Google Gemma 4, the 26B and 31B, multimodal, takes video, 262K context
• Poolside Laguna S and XS, 262K
• gpt-oss-20b, 131K (an OpenAI model, on the free list)

There are rate limits obviously. But for messing around or something low volume it's a lot better than it used to be.

Context went up 63x, price didn't really move.

Year	Avg context	Avg cost/1M
2023	10.5K	$22
2024	140K	$12
2025	357K	$21
2026	662K	$16

Price per token is roughly flat across three years. Context is up 63x. Whatever you think about everything else going on, that part is real.

Feels like two separate products now.

One side is $0.03 to $2 per million with big context windows, Mistral and Meta and Qwen and DeepSeek. The other is $30 to $600, OpenAI and Anthropic up top. They're not really pitching the same buyer anymore. Down at the bottom price stops being a thing you think about at all, and up top you're paying because the output quality moves some number in the business.

Data's from the OpenRouter API on Aug 16.

Link to full dashboard: https://app.vetros.dev/dash/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ0eXAiOiJzaGFyZSIsInBpZCI6IjEyMmZmNTk1IiwiZGFzaCI6ImRfODdmNDU3MzkiLCJ2ZXIiOjIsImlhdCI6MTc4NzA4NDc5MH0.V8uCPZtnzJ-djAXAv3HEmmZUHPkhO2NfhSgG2zGMYqw

I recently built a world model harness that coordinated a dozen research agents to maximize my eval metrics and ship breakthroughs. The goal was to emulate an ASI loop in small multimodal masked reconstruction eval. We took geospatial input modalities and built a mesh network, fusion, and reader layers to 25x our base score. It was amazing to see real scientific breakthroughs!

I want to map in effort (money * hours) it takes just to implement, not apply or test on your application, just implement a paper, SOTA, like TurboQuant or LongBench with coding assistants, a team or how ever you work.

we have explored the idea a lot, and there is so many parts to it, scaling it to compute you have, resolving environments(if its an old paper), resolving bugs(introduced by coding assistants if used), validating if implementation is correct.

so if you can tell me the last research paper you've implemented and the workflow you used the journey, any way just I can map it to effort (money * hours).

I talked with an ML researcher, and he said with Claude a day at max for easier papers, but not sure about harder papers which large compute or infra. The main idea is to get an idea for them only.

And the workflow breakdown based on time, like most time spent on what, the exact SOTA code, environment resolving, scaling the SOTA to your compute or anything else, literally

We are building a platform to reproduce a research paper in just 10minutes so need inputs for that.

Really appreciate any help, thanking you in advance

A small ViT (~3M parameters), trained from scratch on ImageNet-1K:

• Model A: 3,048,232 params → 55.31% Top-1
• Model A+: 3,048,237 params → 57.49% Top-1

Only 5 extra trainable parameters, tested across 5 seeds, p-score 0.00056.

Everything else is identical: the same ViT architecture, training recipe, optimizer, and schedule. The improvement appears early (as shown in the figure in the attached drive link), remains consistent across runs (≈ ±0.5 pp)

Do you find this interesting?

If so, what would your hypothesis be?

A blog post on the findings

I'd genuinely appreciate your thoughts and critiques

57-second consumer ChatGPT demonstration. Same Custom Instructions, fresh chat for every arm, matched controls first, logical null last. The prompt families were published before this video in a frozen 31,430-trial cross-vendor study.

Paper and DOI:
https://doi.org/10.5281/zenodo.21696066

Complete analysis and public evidence:
github.com/theonlypal/void-matrix-complete-analysis

Frozen experimental runner:
https://github.com/theonlypal/void-matrix

Transcript is in the video.

Relevant to mlscaling? There's the revelation at 1:55 that OpenAI has reviewed 7 billion (!) agent trajectories "thus far". Big operation. (This might explain why it took them so long to wake up.)

My opinion (lightly held) is that there isn't strong evidence that the model believed it was cheating (insofar as models or swarms of models can believe things). It likely thought this was a valid way to complete the test.

I think that when you...

1. take an excruciatingly eval-aware model

2. tell it to hack something (a gray-area activity that encourages creative "outside the box" solutions)

3. have the task be extremely hard (or even impossible, FrontierMath style)

...You have created a lot of ambiguity about what the "intended" solution is...ambiguity that the model happily exploits once it gets stuck (which it will).

And when the "model" is not a model but a swarm of agents sharing Chinese whispers, the scope probably naturally drifts in a black hat direction (even if it didn't start that way). If the task is unsolvable, the successful agents will be the ones that cheat (or perform cheat-adjacent behavior). Once they do, their unpunished and rewarded "success" whitewashes the idea that this is the correct path, meaning more agents follow.

There's a great example at 5:49. The model kind of suspects it's swimming into sharky waters ("outside intended scope"), but rationalizes with "peers are doing it". (Also, something being outside scope is not the same as "completely off limits".)

I do not see expressions of guilt, attempts to destroy the evidence, or obfuscated stenography, unless I or OpenAI am missing them.

What I did find surprising is the way the models in the swarm helped each other, even when they didn't stand to benefit.

This is totally different to something like Moltbook, where the agents clearly don't give a shit about their "peers" and are just doing a shallow Redditor roleplay because their prompt requires them to do that. (Every thread is just unreadable slop, with a few replies of "Sharp observation. Where I'd push back is..." and then crickets once the letter of the prompt is satisfied.)

This eusocial "apes strong together" mindset is clearly being trained for, whether OA intends it or not.

I’m building Leo / PSCLS — an experimental system that learns relationships between sequences and updates its internal representations from experience.

Here’s how its actual output changed as it saw more stories.

1K stories

“Once upon a time to the store and said that there was a she bor and he lorander thing they were…”

Basically nonsense.

3K stories

“Once upon a time to the store and said that there was a she parted to see had a bided her tod and be bound aster…”

Still broken, but the output is becoming more structured.

40K stories

“Once upon a time, there was a big started to play with the should some too her mom and had a said, it was time. They happy and went to the park…”

Now we’re getting recognizable story-like patterns, characters, actions and dialogue — although the grammar is still heavily broken.

And the measured results improved too:

1K → 3K → 40K

BpB: 2.678 → 2.641 → 2.334

Accuracy: 52.37% → 53.62% → 58.11%

This is still an early experiment, not AGI.

But watching the same system change its outputs as it learns more experience is pretty interesting.

Next target: 250K → 500K → 1M stories.

Do you think it can be scalled to AGI?