The AI Superforecasters Are Here

Scott Alexander reports from the annual prediction market conference on the emergence of AI superforecasters — frontier models augmented with specialized "scaffolds" that guide them through extensive research processes to produce calibrated probability estimates.

He tests FutureSearch on a question about halving cold rates by 2040. Within five minutes, the AI deployed three subagents, read 16 websites, cited 212 sources, and returned a 7% probability — at a cost of $8. A human superforecaster independently gave 5–10%.

Metaculus data shows AI approaching community-prediction accuracy but still trailing top professional superforecasters. Scaffolded forecasting systems add about nine months of progress beyond base models. In the recent Metaculus Cup, humans took first and second but an AI (Preseen) placed third — a "statistical dead heat." Metaculus forecasters give a 95% chance an AI wins before 2030.

Alexander identifies three distinctive advantages of AI forecasters: perceived legitimacy (people trust machine probabilities more readily), standardized branding (one AI could achieve Nate Silver's reputation but scale to everyone), and no adversarial incentives (unlike prediction markets, AIs don't game resolution criteria).

He argues that superforecasting could serve as the "opinion layer" of AI — allowing calibrated probabilities on controversial topics rather than refusals to engage. Prediction markets retain value as canonical arbiters between competing AI forecasts. He cautions, however, that he caught himself rejecting an AI forecast (a 1–2.2% US-China AI treaty probability) the first time it genuinely challenged his worldview.