title: Pitfalls in Evaluating Language Model Forecasters
summary: Large language models (LLMs) have recently been applied to forecasting tasks,
  with some works claiming these systems match or exceed human performance. In
  this paper, we argue that, as a community, we should be careful about such
  conclusions as evaluating LLM forecasters presents unique challenges. We
  identify two broad categories of issues: (1) difficulty in trusting evaluation
  results due to many forms of temporal leakage, and (2) difficulty in
  extrapolating from evaluation performance to real-world forecasting. Through
  systematic analysis and concrete examples from prior work, we demonstrate how
  evaluation flaws can raise concerns about current and future performance
  claims. We argue that more rigorous evaluation methodologies are needed to
  confidently assess the forecasting abilities of LLMs.
paper_authors._id: 683ee251668eea64b040a050
paper_authors.name: Daniel Paleka
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-06-03T12:43:51.277Z
paper_authors._id: 62363277cd8e1f47c865190a
paper_authors.avatarUrl: /avatars/fb42d42dedd4dde8df4109883cbe7ea9.svg
paper_authors.fullname: Daniel Paleka
paper_authors.type: user
paper_authors.user: dpaleka
paper_authors._id: 683ee251668eea64b040a051
paper_authors.name: Shashwat Goel
paper_authors._id: 683ee251668eea64b040a052
paper_authors.name: Jonas Geiping
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-06-03T12:44:04.836Z
paper_authors._id: 63d86dbf3130cadcaf8bdd11
paper_authors.avatarUrl: /avatars/29d79a0c6dcec01111ef192fecd0fa7a.svg
paper_authors.fullname: Jonas Geiping
paper_authors.type: user
paper_authors.user: JonasGeiping
paper_authors._id: 683ee251668eea64b040a053
paper_authors.name: Florian Tramèr
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-06-03T12:44:13.091Z
paper_authors._id: 63568f18ba90b4ea9fe91cb5
paper_authors.avatarUrl: /avatars/3e8b3c573e20cf80d329a312bfc34728.svg
paper_authors.fullname: Florian Tramer
paper_authors.type: user
paper_authors.user: ftramer
paper_title: Pitfalls in Evaluating Language Model Forecasters
paper_summary: Large language models (LLMs) have recently been applied to forecasting tasks,
  with some works claiming these systems match or exceed human performance. In
  this paper, we argue that, as a community, we should be careful about such
  conclusions as evaluating LLM forecasters presents unique challenges. We
  identify two broad categories of issues: (1) difficulty in trusting evaluation
  results due to many forms of temporal leakage, and (2) difficulty in
  extrapolating from evaluation performance to real-world forecasting. Through
  systematic analysis and concrete examples from prior work, we demonstrate how
  evaluation flaws can raise concerns about current and future performance
  claims. We argue that more rigorous evaluation methodologies are needed to
  confidently assess the forecasting abilities of LLMs.
paper_ai_summary: Evaluation of large language models for forecasting is flawed due to temporal leakage and difficulties in translating performance to real-world forecasting, necessitating more rigorous methodologies.
paper_ai_keywords: large language models forecasting temporal leakage evaluation methodologies