Evaluating LLMs on Real-World Forecasting Against Human Superforecasters
Large language models (LLMs) have demonstrated remarkable capabilities across
diverse tasks, but their ability to forecast future events remains
understudied. A year ago, large language models struggle to come close to the
accuracy of a human crowd. I evaluate state-of-the-art LLMs on 464 forecasting
questions from Metaculus, comparing their performance against human
superforecasters. Frontier models achieve Brier scores that ostensibly surpass
the human crowd but still significantly underperform a group of
superforecasters.
686ca9e2364e2ad167eb5461
Janna Lu
Evaluating LLMs on Real-World Forecasting Against Human Superforecasters
Large language models (LLMs) have demonstrated remarkable capabilities across
diverse tasks, but their ability to forecast future events remains
understudied. A year ago, large language models struggle to come close to the
accuracy of a human crowd. I evaluate state-of-the-art LLMs on 464 forecasting
questions from Metaculus, comparing their performance against human
superforecasters. Frontier models achieve Brier scores that ostensibly surpass
the human crowd but still significantly underperform a group of
superforecasters.
State-of-the-art large language models underperform human superforecasters in forecasting accuracy despite achieving Brier scores that surpass the human crowd.
large language models LLMs forecasting Metaculus Brier scores superforecasters