title: MIRAI: Evaluating LLM Agents for Event Forecasting
summary: Recent advancements in Large Language Models (LLMs) have empowered LLM agents
  to autonomously collect world information, over which to conduct reasoning to
  solve complex problems. Given this capability, increasing interests have been
  put into employing LLM agents for predicting international events, which can
  influence decision-making and shape policy development on an international
  scale. Despite such a growing interest, there is a lack of a rigorous benchmark
  of LLM agents' forecasting capability and reliability. To address this gap, we
  introduce MIRAI, a novel benchmark designed to systematically evaluate LLM
  agents as temporal forecasters in the context of international events. Our
  benchmark features an agentic environment with tools for accessing an extensive
  database of historical, structured events and textual news articles. We refine
  the GDELT event database with careful cleaning and parsing to curate a series
  of relational prediction tasks with varying forecasting horizons, assessing LLM
  agents' abilities from short-term to long-term forecasting. We further
  implement APIs to enable LLM agents to utilize different tools via a code-based
  interface. In summary, MIRAI comprehensively evaluates the agents' capabilities
  in three dimensions: 1) autonomously source and integrate critical information
  from large global databases; 2) write codes using domain-specific APIs and
  libraries for tool-use; and 3) jointly reason over historical knowledge from
  diverse formats and time to accurately predict future events. Through
  comprehensive benchmarking, we aim to establish a reliable framework for
  assessing the capabilities of LLM agents in forecasting international events,
  thereby contributing to the development of more accurate and trustworthy models
  for international relation analysis.
paper_authors._id: 66836a75f771146afb3b4486
paper_authors.name: Chenchen Ye
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-02T07:39:01.011Z
paper_authors._id: 661e2aac940272b8edc5a5dd
paper_authors.avatarUrl: /avatars/000f7c8fae4ed38af3c616135a3e8d2d.svg
paper_authors.fullname: Chenchen Ye
paper_authors.type: user
paper_authors.user: chenchenye
paper_authors._id: 66836a75f771146afb3b4487
paper_authors.name: Ziniu Hu
paper_authors._id: 66836a75f771146afb3b4488
paper_authors.name: Yihe Deng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-02T07:38:58.926Z
paper_authors._id: 642f4c789b2484d7d8551a93
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/642f4c789b2484d7d8551a93/0lH4YXcbZa-Xlzj6ESo7F.jpeg
paper_authors.fullname: Yihe Deng
paper_authors.type: user
paper_authors.user: ydeng9
paper_authors._id: 66836a75f771146afb3b4489
paper_authors.name: Zijie Huang
paper_authors._id: 66836a75f771146afb3b448a
paper_authors.name: Mingyu Derek Ma
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-02T07:38:56.904Z
paper_authors._id: 644c1df68c51ddbe0ea6ddd7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/644c1df68c51ddbe0ea6ddd7/KAsOhnOSbrOK54OHtJpiQ.jpeg
paper_authors.fullname: Mingyu Derek Ma
paper_authors.type: user
paper_authors.user: derekma
paper_authors._id: 66836a75f771146afb3b448b
paper_authors.name: Yanqiao Zhu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-02T07:38:54.633Z
paper_authors._id: 65dda5295eea01ee95e21f64
paper_authors.avatarUrl: /avatars/f935e7db6038661bef2305b7aeda8c9e.svg
paper_authors.fullname: Yanqiao Zhu
paper_authors.type: user
paper_authors.user: sxkdz
paper_authors._id: 66836a75f771146afb3b448c
paper_authors.name: Wei Wang
paper_title: MIRAI: Evaluating LLM Agents for Event Forecasting
paper_summary: Recent advancements in Large Language Models (LLMs) have empowered LLM agents
  to autonomously collect world information, over which to conduct reasoning to
  solve complex problems. Given this capability, increasing interests have been
  put into employing LLM agents for predicting international events, which can
  influence decision-making and shape policy development on an international
  scale. Despite such a growing interest, there is a lack of a rigorous benchmark
  of LLM agents' forecasting capability and reliability. To address this gap, we
  introduce MIRAI, a novel benchmark designed to systematically evaluate LLM
  agents as temporal forecasters in the context of international events. Our
  benchmark features an agentic environment with tools for accessing an extensive
  database of historical, structured events and textual news articles. We refine
  the GDELT event database with careful cleaning and parsing to curate a series
  of relational prediction tasks with varying forecasting horizons, assessing LLM
  agents' abilities from short-term to long-term forecasting. We further
  implement APIs to enable LLM agents to utilize different tools via a code-based
  interface. In summary, MIRAI comprehensively evaluates the agents' capabilities
  in three dimensions: 1) autonomously source and integrate critical information
  from large global databases; 2) write codes using domain-specific APIs and
  libraries for tool-use; and 3) jointly reason over historical knowledge from
  diverse formats and time to accurately predict future events. Through
  comprehensive benchmarking, we aim to establish a reliable framework for
  assessing the capabilities of LLM agents in forecasting international events,
  thereby contributing to the development of more accurate and trustworthy models
  for international relation analysis.
paper_ai_summary: MIRAI introduces a benchmark to evaluate LLM agents' forecasting capabilities for international events using historical databases and APIs.
paper_ai_keywords: Large Language Models (LLMs) GDELT event database temporal forecasting forecasting horizons relational prediction tasks domain-specific APIs tool-use historical knowledge international relation analysis