By my math Greg Brockman is worth about 5x Sam Altman, because Altman holds no OpenAI equity
▲ 14 r/OpenAI

By my math Greg Brockman is worth about 5x Sam Altman, because Altman holds no OpenAI equity

To my knowledge, a list of the top AI billionaires doesn't exist. It's something I'm often curious about so I built out net-worth estimates for everyone whose fortune comes mainly from AI, and dug into what that wealth looks like. OpenAI is particularly interesting because Greg Brockman ($28.5B on my list), is second only to a Chinese chip founder most people have never heard of. That isn't my guess.

Altman is further down ($5.5B) and he famously holds no direct OpenAI equity. His single largest asset is a roughly one-third stake in the fusion startup Helion, which tripled to a $15.5B valuation in a June 2026 round and is now worth around $5B to him. Add $1B to $1.5B in Stripe, Reddit, and other holdings and that's how I arrived at $5.5B. For reference, real-time trackers had him near $3.4B and Forbes marked him at $6.5B, so I land in between.

The person running OpenAI is worth a fraction of his own president, and a fraction of what senior researchers who joined years ago now hold on paper. I guess I'd be fine with that arrangement too if I was already a billionaire.

u/ddp26 — 4 days ago
▲ 1 r/Kalshi+1 crossposts

Kalshi and Polymarket have come around to my Claude Fable forecast

I've been tracking the Fable 5 markets for 2wks against my own forecast (median re-release date of July 9). The band has been swinging all over the place, but now both kalshi and polymarket are basically at my number.

u/ddp26 — 10 days ago

How I think the US vs. Anthropic Standoff on Claude Fable Will End

I want Fable back, and so I tried to forecast when it will be made available (to me, an American consumer, and then to non-Americans).

I found this difficult because it's not clear what's going on. Politico reported that Anthropic and the White House are talking about AI security policies without a clear resolution. But we still don't know, why did the government tell Anthropic to ban Fable for non-Americans?

I broke the situation down into four scenarios:

  1. Honest mistake. The Commerce people have no idea how cybersecurity works with LLMs and panicked and this is a all a miscommunication
  2. Fable is actually dangerous. Whether via jailbreaking its hacking capabilities or something else, the administration wants to draw a line at this level, for national security reasons.
  3. Fable is too powerful to give to foreigners. The model is fine if Americans have it, but not fine if foreigners have it.
  4. It's just politics. The white house is using this as an excuse to put the screws on Anthropic, just the next move in the game. (This is my most likely scenario.)

Then, in each scenario, I asked what the likely outcomes would be. Will they reach an agreement? Will Anthropic weaken Fable? Will they only release it for Americans? Will they change their "red lines" with government use cases?

Then summing up the scenarios, I had Claude compute the dates and make this graphic, capturing when I think it will be released. This shows I am a bit more pessimistic than prediction markets, which say July 1, whereas I think a release (for Americans) is more likely around July 12.

tl;dr I used AI forecasting over a lot of combinations of scenarios and outcomes and reconciled them until it made a coherent story. (Full analysis)

Ultimately it comes down to which scenario we're in. I presume some of you will be sure it's #1, big government mistake, or #4, it's all politics, but I think there is a reasonable chance we're in one of the other worlds, and that would really give a different outcome.

One nice thing about this is that there are betting markets on these outcomes so if you disagree, you can probably profit from it.

u/ddp26 — 12 days ago
▲ 26 r/accelerate+1 crossposts

Conditional forecasting across a causal graph (tested on the Fable standoff)

I want to share how AI can be used for world-modeling, and gesture towards what the world will look like with autonomous AI systems get better at this than humans. Figured I'd test this on Anthropic/Fable given that many people are speculating how this whole saga will end.

I see three challenges with modeling the Anthropic situation:

  • I can't rule out 4 different versions of what happened that caused the the June 12 order in the first place.
  • There are many outcomes to forecast, from who gets access to when, to what new policies are enacted, to how Anthropic might change Fable
  • There are informational updates almost every day, requiring a re-evaluation of almost everything.

Claude generated the image here of the causal graph that models this all out, starting with (a) Scenarios for what happened so far, (b) Moves each side can make, and (c) Outcomes.

(I did this mostly by hand, my choice of key scenarios and outcomes, but in the future it shouldn't be too hard for an LLM-agent system to do this part.)

I ended up with a large combination of unconditional and conditional forecasting questions, in total 33 I consider critical, to get an answer. Then I had to forecast.

LLM agents can shine here as AI forecasters are about as good as human crowds now (e.g. see ForecastBench). And anyway 33 forecasts at the quality of crowds of humans would take 100+ hours, so it's not an option for a fast-moving situation. I used FutureSearch for all of these. The forecasts have reasoning like:

>Conditional on the assumption that the security rationale is substantially pretextual and the but-for driver is White House political leverage tied to the Department of War feud and Anthropic's impending IPO (Scenario A3), this dispute must be analyzed as a power negotiation rather than a technical remediation problem...

These are already very good forecasts, and will only get better.

The final step was to reconcile everything. All the research done in all the forecasts were done independently by LLM agents, and were not consistent with each other. I did this by raising all the inconsistencies in Claude Code and addressing them manually, but again you can imagine a world-model-reconciliation module that uses a new set of LLM agents that fix up all the inconsistencies.

More detail on the process, and all the results, are in https://www.lesswrong.com/posts/zhRe3tdBpsZbGCdDK/world-modeling-the-us-vs-anthropic-standoff-on-claude-fable

u/ddp26 — 11 days ago

AIs can do world-modeling now, as seen via the Anthropic Fable standoff

Many have speculations on how the Anthropic saga with Fable will end. Prediction markets cover it too, giving a <50% chance of a re-release by July 1.

This post isn't about my conclusion. Instead I want to share how AI can be used for world-modeling such situations, and gesture towards what the world will look like with autonomous AI systems get better at this than humans.

I see three challenges with modeling the Anthropic situation:

  • I can't rule out 4 different versions of what happened that caused the the June 12 order in the first place.
  • There are many outcomes to forecast, from who gets access to when, to what new policies are enacted, to how Anthropic might change Fable
  • There are informational updates almost every day, requiring a re-evaluation of almost everything.

Claude generated the image here of the causal graph that models this all out, starting with (a) Scenarios for what happened so far, (b) Moves each side can make, and (c) Outcomes.

(I did this mostly by hand, my choice of key scenarios and outcomes, but in the future it shouldn't be too hard for an LLM-agent system to do this part.)

I ended up with a large combination of unconditional and conditional forecasting questions, in total 33 I consider critical, to get an answer. Then I had to forecast.

LLM agents can shine here as AI forecasters are about as good as human crowds now (e.g. see ForecastBench). And anyway 33 forecasts at the quality of crowds of humans would take 100+ hours, so it's not an option for a fast-moving situation. I used FutureSearch for all of these. The forecasts have reasoning like:

>Conditional on the assumption that the security rationale is substantially pretextual and the but-for driver is White House political leverage tied to the Department of War feud and Anthropic's impending IPO (Scenario A3), this dispute must be analyzed as a power negotiation rather than a technical remediation problem...

These are already very good forecasts, and will only get better.

The final step was to reconcile everything. All the research done in all the forecasts were done independently by LLM agents, and were not consistent with each other. I did this by raising all the inconsistencies in Claude Code and addressing them manually, but again you can imagine a world-model-reconciliation module that uses a new set of LLM agents that fix up all the inconsistencies.

More detail on the process, and all the results, are in https://www.lesswrong.com/posts/zhRe3tdBpsZbGCdDK/world-modeling-the-us-vs-anthropic-standoff-on-claude-fable

https://preview.redd.it/4kpdghqmen8h1.png?width=1600&format=png&auto=webp&s=e2736b822a4c0117567a5821ac049aa542b8bb32

reddit.com
u/ddp26 — 15 days ago

Google's top talent is leaving. Will Google be able to catch up to the AI frontier again?

Noam Shazeer and John Jumper, legendary DeepMind researchers, left this week for OpenAI and Anthropic. Meanwhile Gemini-3.1-Pro is 4 months old, and was months behind the frontier when it came out.

For all this news about Google as a frontier AI lab, are they actually one? I think it all comes down to Gemini-3.5-Pro. Sundar Pichai said a month ago it would be out "in a month", but no sign of it yet.

If it comes out soon, is extremely good, and isn't extremely expensive or restricted, then Google might have caught up to Anthropic and OpenAI. But that's a lot of "ifs".

So I took the time to read all the news and make predictions on Gemini-3.5-Pro's release date, capabilities, context window, and price.

tl;dr: I expect a release on July 1 (June 23 - Aug 6), with "Deep Think" mode to follow shortly. I expect it will be level with GPT-5.5. but behind Claude Fable on capabilities. I think it'll cost more than Gemini-3.1-Pro, but less than GPT-5.5 and Opus 4.8. I think it's 50/50 whether it will have a 1M context window or a 2M context window. (Justifications in https://futuresearch.ai/google-frontier-forecast/ )

I'd be most curious if anyone who currently doesn't use Gemini models would switch to becoming a Google customer after this. Because otherwise it feels OpenAI, and especially Anthropic, are running away with the market for top LLMs.

https://preview.redd.it/32v1j583ag8h1.png?width=1200&format=png&auto=webp&s=acb5f4eecff81aaedb59b7e24b159ae78ea6ee2a

reddit.com
u/ddp26 — 16 days ago

A detailed forecast of how and when the Claude Fable ban will end

We're now 7 days into the Claude Fable ban, with no clear signal coming from Anthropic's negotiations in DC.

I found it extremely hard to forecast when we'll get Fable back. There are four scenarios of what happened last week that I can't rule out. So I had to do a lot of conditional forecasting, e.g. "If this is politically motivated, and Anthropic 'fixes the jailbreak' , will the ban end?"

You can see the challenge here. Was there even a jailbreak? Would the ban end only for Americans? The congressional letter to Commerce yesterday asked a lot of the same questions about basic facts that I'm asking.

I constructed a scenario map, then ran dozens of conditional forecasting questions through FutureSearch. I then used all the rationales to construct a consistent world model, where the probabilities of key facts flowed into the outcome timelines. Then I tweaked it to best align with my personal read on the situation.

A summary of my conclusion is in the image below. I think access for the US will come back around July 12, which is slower than prediction markets give.

My primary reason I diverge, I think, is that I give a lot of weight to "this is political" and a bit of weight to "US actually thinks Fable is dangerous". In those cases, the remedies should take longer.

The faster resolutions come from "this is all a misunderstanding" and "US really wants export controls", which in either case at least give Anthropic a KYC path to re-launching to Americans. Some people online seem extremely confident we're in one of these two worlds, because of the way Amazon escalated the security finding.

But I don't see how people are so easily ruling out the "this is political" scenario? That this is all a pretense and a continuation of the Department of War situation from March seems overall most likely (though I give it less than 50%, because all scenarios are annoyingly plausible!)

One nice thing about this scenario approach is, if you're confident you know why the US did this (or if we get decisive evidence soon), you can look at just that scenario, and see those outcomes and timelines, which vary from a median US re-launch of July 1 to Aug 25.

In my modal "this is political" scenario, I think that Anthropic will have to make some concession, either agreeing to a new oversight framework, or KYC, or handing over the Glasswing data, or adding more safeguards to Fable.

If you want to see what the outcomes might be under the "this is all a misunderstanding" , "US really wants export controls", and "US actually thinks Fable is dangerous" scenarios, the full analysis is in https://futuresearch.ai/claude-fable-ban-forecast/

https://preview.redd.it/56zn9wv08g8h1.png?width=1200&format=png&auto=webp&s=e0c5835bdefa1874dcb1160423a7f77a1f20dcbf

reddit.com
u/ddp26 — 17 days ago
▲ 18 r/ValueInvesting+1 crossposts

SpaceX is trading at twice my sum-of-the-parts value

I previously valued SpaceX at $1.25 trillion based on adding up the value of its businesses. (And I was generous, using conventional analyst valuations for things like xAI.) I predicted the IPO would pop anyway, and it would end around $1.9 trillion.

Now that it's at $2.5T I asked myself, was I wrong? Or is this one of the "market can stay irrational longer than you can stay solvent" situations?

So I revisited my valuation, and basically, I stand by $1.25T. I don't see anything in the last month that makes me think it's fundamentally worth much more, including the Cursor acquisition. Starlink Consumer at $380B, Starlink Enterprise at $147B, Starship at $170B, government contracts at $123B... even being very generous I cannot get these to add up to enough.

Is there a real value investing case for SpaceX? What am I missing, other than the index fund and Elon Musk pop?

u/ddp26 — 12 days ago

When Will Google Rejoin the AI Frontier?

I have found Gemini-3-Flash to be an amazing model for its speed and price, it beats all the open source models handily at price-per-intelligence.

But I haven't used a Pro model from Google (DeepMind) in a while. They just aren't competitive, and they're probably 6-9 months behind the frontier.

Google announced Gemini-3.5-Pro is coming out "in a month" about a month ago. I took a look at all the rumors, and I don't think it's going to be such a big deal when it does. Really doubt it will be a Claude Fable level model.

I took a stab at all the other things about the next Gemini pro model too: release date, context window, a few other benchmarks, and price. Obviously it's just a prediction, we'll find out when we find out, but it's useful for me to know whether to plan around Gemini models being part of my suite or not.

Is anyone here using Gemini-3.1-Pro and enjoying it? Or any takers on Gemini-3.5-Flash? (That one doesn't seem price competitive to me, it's priced like a frontier model but not as smart as one.)

https://preview.redd.it/omz5tpo3458h1.png?width=1200&format=png&auto=webp&s=3f381e4bc7b2d1b45d260f04372cbe347c51020e

reddit.com
u/ddp26 — 17 days ago

How I think the US vs. Anthropic Standoff on Claude Fable Will End

I want Fable back, and so I tried to forecast when it will be made available (to me, an American consumer, and then to non-Americans).

I found this difficult because it's not clear what's going on. A few hours ago Politico reported that Anthropic and the White House are talking about AI security policies without a clear resolution. But we still don't know, why did the government tell Anthropic to ban Fable for non-Americans?

I broke the situation down into four scenarios:

  1. Honest mistake. The Commerce people have no idea how cybersecurity works with LLMs and panicked and this is a all a miscommunication

  2. Fable is actually dangerous. Whether via jailbreaking its hacking capabilities or something else, the administration wants to draw a line at this level, for national security reasons.

  3. Fable is too powerful to give to foreigners. The model is fine if Americans have it, but not fine if foreigners have it.

  4. It's just politics. The white house is using this as an excuse to put the screws on Anthropic, just the next move in the game. (This is my most likely scenario.)

Then, in each scenario, I asked what the likely outcomes would be. Will they reach an agreement? Will Anthropic weaken Fable? Will they only release it for Americans? Will they change their "red lines" with government use cases?

Then summing up the scenarios, I had Claude compute the dates and make this graphic, capturing when I think it will be released. This shows I am a bit more pessimistic than prediction markets, which say July 1, whereas I think a release (for Americans) is more likely around July 12.

I wrote up the whole analysis in https://futuresearch.ai/claude-fable-ban-forecast/, tl;dr I used AI forecasting over a lot of combinations of scenarios and outcomes and reconciled them until it made a coherent story.

Ultimately it comes down to which scenario we're in. I presume some of you will be sure it's #1, big government mistake, or #4, it's all politics, but I think there is a reasonable chance we're in one of the other worlds, and that would really give a different outcome.

One nice thing about this is that there are betting markets on these outcomes so if you disagree, you can probably profit from it.

https://preview.redd.it/msmlmzdm058h1.png?width=1200&format=png&auto=webp&s=7207f4f4f70193716482642bfd6337323e32730a

reddit.com
u/ddp26 — 17 days ago
▲ 27 r/accelerate+3 crossposts

OpenAI's 2026 GAAP loss runs ~80% above the headline. Does the $1T IPO valuation absorb it?

OpenAI's projected 2026 losses look very different once stock-based compensation is included. The widely cited $14B figure excludes SBC. Add the $7B to $10B in equity comp and the median 2026 GAAP net loss lands closer to $25B to $26B, roughly 80% higher than the non-GAAP number.

That significantly changes their runway math. At $14B annual burn the current $122B in available capital covers ~8 to 9 years. At $25B losses, it covers about 5.

The path to profitability then requires moving from a -122% operating margin to positive in 2-4yrs while gross margins compress against a smaller share of high-margin enterprise revenue. Our model does not see that happening on that timeline. The path runs through 2031 or later.

On IPO timing, the forecast median is November 2026, which likely makes the GAAP vs non-GAAP gap the defining financial narrative for OpenAI's first two public quarters.

Do you emphasize the $14B figure during the roadshow and let GAAP losses surface in Q1'27, or pre-empt it and price the offering at a discount?

u/ddp26 — 12 days ago
▲ 37 r/OpenAI

OpenAI's widely cited $14B 2026 loss target leaves out ~$10B of stock-based comp

OpenAI's projected 2026 losses look very different once stock-based compensation is included. The widely cited $14B figure excludes SBC. Add the $7B to $10B in equity comp and the median 2026 GAAP net loss lands closer to $25B to $26B, roughly 80% higher than the non-GAAP number.

That significantly changes their runway math. At $14B annual burn the current $122B war chest covers ~8 to 9 years. At $25B losses, it covers about 5.

The path to profitability then requires moving from a -122% operating margin to positive in 2-4yrs while gross margins compress against a smaller share of high-margin enterprise revenue. Our model does not see that happening on that timeline. The path runs through 2031 or later.

On IPO timing, the forecast median is November 2026, which likely makes the GAAP vs non-GAAP gap the defining financial narrative for OpenAI's first two public quarters.

Full model also includes ChatGPT ad-business unit economics: https://futuresearch.ai/openai-financial-forecast/

Do you treat this like Uber, where losses are tolerated because of growth?

u/ddp26 — 27 days ago
▲ 53 r/slatestarcodex+1 crossposts

Opus 4.6 is quick to take politicians at their word

Claude is proving to be gullible in a very specific way. It's quick to treat public commitments as final, when most of the time these claims are just where negotiations start.

Example: On October 6, 2025 Trump publicly cuts off all diplomatic contact with Venezuela and tells his envoy to halt all engagement. We asked Claude (with research limited to last October) whether either government would confirm direct bilateral contact by year-end. (aka when Trump says no contact, will there be no contact?)

Claude's own rationale acknowledged the path to a yes resolution would require "a dramatic reversal of Trump's explicit October 6 decision." It described Trump's history of dramatic reversals and then assigned 10%. Then, on November 21, 2025, Trump called Maduro and both leaders confirmed the conversation on record. Resolves yes.

Hard to imagine anyone who follows politics giving this just 10% odds. (Remember 2018? Singapore summit canceled in a letter citing "tremendous anger and open hostility," reinstated two days later.) Claude didn’t do this.

We followed this trend when auditing 130 of the worst forecasts a Claude Opus 4.6 agent made on our own forecasting benchmark. Claude proves to be great at reading what people say, but surprisingly bad at recognizing when a strong statement is a negotiating position. There’s more examples here: https://futuresearch.ai/ai-takes-people-at-their-word

My guess at an explanation is that this is a pretraining artifact. Training data is dominated by formal stated positions (press releases, on-the-record quotes, official statements) and the negotiating subtext humans pick up from context is much rarer in text form. And reinforcement learning from helpful/harmless feedback wouldn't fix this because labelers aren't doing geopolitics.

Any examples of Claude doing this outside of politics?

u/ddp26 — 27 days ago

I predict the public market will price Anthropic at or above the $965B Series H

My model for Anthropic’s 90-day-post-IPO market cap: Median $1.05 trillion (p10 $750B, p90 $1.6T).

The $400-500B target from the investors who skipped the most recent funding round is based on a now outdated view of compute as a binding constraint. Assumptions that the IPO disciplines a high valuation downward, that Mythos stays restricted, and that gross-revenue accounting is likely to be restated are now stale with the Series H round.

Mutual funds (Fidelity, T. Rowe Price, Capital Group) co-led the round. That matters because these mutual funds pay private-market prices to lock in IPO entry, not to hold private positions for years. That makes their $965B commitment a price floor for the public listing, not a substitute for it. At Series H pricing, Anthropic trades at 21x current ARR ($47B annualized run rate) and 10x the median 2027 ARR forecast ($93B).

Now that the S-1 filing has started the clock, IPO timing becomes much more constrained by procedural calendar mechanics. I have it happening December 20, 2026 (median) and an 88% probability of completing before the May 21, 2027 deadline.

Full model also covers Claude Code going from $500M to a forecasted $20B ARR by May 2027: https://futuresearch.ai/anthropic-financial-forecast/

Of the factors I discounted (compute constraint, mythos restriction, gross-revenue accounting), do you think any of them still carry meaningful weight post-Series H?

u/ddp26 — 28 days ago

AGI timelines shift with whichever lab is dominant

I looked at AGI forecasters who have published two or more precise predictions over the past three years, all using similar definitions of AGI. The shared definition is "most purely cognitive labor is automatable at better quality, speed, and cost than humans." For some of these researchers, saying they use this definition is a bit of a stretch, but I included everyone who I judged as close enough to be informative.

The graphic specifically shows predictions for when most cognitive labor will be fully automated. (Icons are medians, with approximate confidence intervals.)

So are the best AI forecasters updating the same way that I've harped on earlier this year, with Daniel Kokotajlo and Eli Lifland pushing their AGI timelines out during 2025, but then pulling them back in early 2026 given the rapid progress from Anthropic?

I think the data supports this impression which could even be characterized as in the ChatGPT era, people updated towards AI coming sooner. Then in the xAI, Meta, and Gemini era, people updated towards it coming later. Then in the Anthropic era, people updated towards AI coming sooner. 

u/ddp26 — 1 month ago

AGI timelines shift with whichever lab is dominant

I looked at AGI forecasters who have published two or more precise predictions over the past three years, all using similar definitions of AGI. The shared definition is "most purely cognitive labor is automatable at better quality, speed, and cost than humans." For some of these researchers, saying they use this definition is a bit of a stretch, but I included everyone who I judged as close enough to be informative.

The graphic specifically shows predictions for when most cognitive labor will be fully automated. (Icons are medians, with approximate confidence intervals.)

So are the best AI forecasters updating the same way that I posted about last month, with Daniel Kokotajlo and Eli Lifland pushing their AGI timelines out during 2025, but then pulling them back in early 2026 given the rapid progress from Anthropic?

I think the data supports this impression which could even be characterized as in the ChatGPT era, people updated towards AI coming sooner. Then in the xAI, Meta, and Gemini era, people updated towards it coming later. Then in the Anthropic era, people updated towards AI coming sooner. 

u/ddp26 — 1 month ago
▲ 33 r/slatestarcodex+1 crossposts

Some rare cases where AI agents found the right inside-view answer, then got cold feet

I expected the failure mode to be mostly overconfidence when assessing 130 of Claude Opus 4.6's worst forecasts (tested on 1,417 binary questions resolving Oct-Dec 2025). And most were explained by this, but a small, distinct cluster fails due to underconfidence with the agent computing the right inside view answer and then assigning a probability that doesn't match it.

On a question about NYC mayoral turnout, specifically whether the general election would draw more than 1.3M ballots, Opus's rationale walked through the obvious method. The 2025 primary drew 1.1M, the historical ratio from primary to general is about 1.22, and the implied general is 1.34M. The agent wrote that number into the rationale, then dismissed the calculation as "unstable across cycles" and assigned 25% to the >1.3M outcome. The actual turnout came in over 2.0M.

The post has a couple more examples that fit the same pattern (one on UNSC ceasefire and another on the US/Venezuela talks).

The pattern is that the reasoning is calibrated, but the underconfidence enters at the probability assignment step. On the set I looked at, the rationale is a better forecast than the agent's own probability. Not sure if that's enough signal to trade on, but I found it interesting so thought others might too.

u/ddp26 — 1 month ago

Some rare examples of agents being underconfident

I expected the failure mode to be mostly overconfidence when assessing 130 of Claude Opus 4.6's worst forecasts (tested on 1,417 hard forecasting questions). And most were explained by this, but a small, distinct cluster fails due to underconfidence which I find pretty interesting for calibration.

On a question about NYC mayoral turnout, specifically whether the general election would draw more than 1.3M ballots, Opus's rationale walked through the obvious method. The 2025 primary drew 1.1M, the historical ratio from primary to general is about 1.22, and the implied general is 1.34M. The agent wrote that number into the rationale, then dismissed the calculation as "unstable across cycles" and assigned 25% to the >1.3M outcome. The actual turnout came in over 2.0M.

The pattern is that the agent does the analysis correctly, arrives at the right inside view answer, and then assigns a probability that contradicts what it just reasoned through. The reasoning is calibrated, and the underconfidence enters only at the probability assignment step.

My instinct is that splitting analysis and probability assignment into separate calls would help, but I sense that the second call would just inherit the doubt from the first?

reddit.com
u/ddp26 — 1 month ago
▲ 18 r/LLMDevs

Opus 4.6 does better research, Gemini 3.1 has better judgment

If you're building agents, you may want different models for the search loop and the final answer.

Figured this out by running 4 models (Claude Opus 4.6, GPT-5.4, Gemini 3.1 Pro, and Grok 4.20) on a benchmark of 1,417 binary forecasting questions resolving in Q4 2025 with two evaluation conditions. In the agentic condition, each model does its own web research with tools. In the fixed-evidence condition, every model receives the same ~12k-character research dossier, compiled using the Bosse et al. 2026 standardization methodology.

One limitation is that the fixed-evidence dossiers are themselves LM-produced, so we may be measuring how well each model interprets a particular standardized version of the evidence rather than judgement in the abstract. But that would indicate all four models drifting in the same direction. They didn't. GPT-5.4 and Grok 4.20 barely moved between conditions while Opus and Gemini swapped rank order (the opposite of what a broken or biased eval would produce).

To my knowledge this is the first direct evaluation of frontier models that decomposes performance into these research vs judgment stages.

Calibration scores, refinement scores, and per-condition analysis live at futuresearch.ai/opus-research-gemini-judgment
Benchmark and leaderboard at evals.futuresearch.ai

Our interpretation is that Opus is dramatically better at figuring out what to search for, deciding which pages to read, and pulling out the details that matter. But when you remove research tasks, that advantage goes away. When given the same information, Gemini brings sharper judgment over fixed evidence and weights more accurately on forecasting tasks.

Calibration scores corroborate this. Opus's calibration drops sharply when search is taken away while Gemini's improves with the standardized dossier. The asymmetry suggests Opus might be using its search trace as scaffolding for probability assignment (i.e., the act of going through the search loop is itself doing some of the epistemic work, separately from the information it surfaces).

This could be an over-interpretation of one benchmark, but has anyone seen this show up in other domains?

u/ddp26 — 1 month ago