How do you measure whether an AI is actually good at predicting the future? The CTO of startup Obside emailed me recently with a fascinating real-world benchmark.
Instead of giving AI models another standardized test, Obside had ChatGPT, Gemini, Claude, Grok, Mistral, DeepSeek, and Kimi bet on World Cup matches using live Polymarket odds.
An hour before kickoff, each model goes into agent mode, researches the teams, injuries, and other public information, then decides how much of its virtual $10,000 bankroll to wager.
When I checked after the semifinals on Thursday, French open-source darling Mistral led the field, followed by OpenAI’s GPT 5.5, and DeepSeek’s V4. Claude Opus 4.8, meanwhile, sat firmly at the bottom, the only model in the red. Maybe Anthropic’s AI is simply too ethical to be a gambler?
Check out the current standings here.
This fun exercise measures something many AI benchmarks can’t: judgment under uncertainty. That’s a theme I’ve explored before. Last year, I wrote about ChatGPT entering a secret forecasting tournament run by economists, where it performed no better than the average human contestant.
Betting on soccer isn’t the same thing, but it’s another clever way to test whether AI can turn online information into profitable predictions when nobody yet knows the answer.
Sign up for BI’s Tech Memo newsletter here. Reach out to me via email at [email protected].
