title: GAIA: a benchmark for General AI Assistants
summary: We introduce GAIA, a benchmark for General AI Assistants that, if solved,
  would represent a milestone in AI research. GAIA proposes real-world questions
  that require a set of fundamental abilities such as reasoning, multi-modality
  handling, web browsing, and generally tool-use proficiency. GAIA questions are
  conceptually simple for humans yet challenging for most advanced AIs: we show
  that human respondents obtain 92\% vs. 15\% for GPT-4 equipped with plugins.
  This notable performance disparity contrasts with the recent trend of LLMs
  outperforming humans on tasks requiring professional skills in e.g. law or
  chemistry. GAIA's philosophy departs from the current trend in AI benchmarks
  suggesting to target tasks that are ever more difficult for humans. We posit
  that the advent of Artificial General Intelligence (AGI) hinges on a system's
  capability to exhibit similar robustness as the average human does on such
  questions. Using GAIA's methodology, we devise 466 questions and their answer.
  We release our questions while retaining answers to 300 of them to power a
  leader-board available at https://huggingface.co/gaia-benchmark.
paper_authors._id: 655ec78fa3e81145767e66a4
paper_authors.name: Grégoire Mialon
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-23T10:41:30.419Z
paper_authors._id: 6458c976f1f5263c2ef0514c
paper_authors.avatarUrl: /avatars/0cab7449293be91a4fa08c23533b1b1a.svg
paper_authors.fullname: Grégoire Mialon
paper_authors.type: user
paper_authors.user: gregmialz
paper_authors._id: 655ec78fa3e81145767e66a5
paper_authors.name: Clémentine Fourrier
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-06-28T19:57:36.463Z
paper_authors._id: 6202a599216215a22221dea9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1644340617257-noauth.png
paper_authors.fullname: Clémentine Fourrier
paper_authors.type: user
paper_authors.user: clefourrier
paper_authors._id: 655ec78fa3e81145767e66a6
paper_authors.name: Craig Swift
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-23T10:41:39.740Z
paper_authors._id: 6467fee96192d391422669d5
paper_authors.avatarUrl: /avatars/595133486c0d73eb21138cbd0a49ae48.svg
paper_authors.fullname: Craig Swift
paper_authors.type: user
paper_authors.user: SwiftyOS
paper_authors._id: 655ec78fa3e81145767e66a7
paper_authors.name: Thomas Wolf
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-23T10:41:53.000Z
paper_authors._id: 5df7e9e5da6d0311fd3d53f9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1583857746553-5df7e9e5da6d0311fd3d53f9.jpeg
paper_authors.fullname: Thomas Wolf
paper_authors.type: user
paper_authors.user: thomwolf
paper_authors._id: 655ec78fa3e81145767e66a8
paper_authors.name: Yann LeCun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-23T10:52:30.928Z
paper_authors._id: 64ed0b8c2203a126eb1a5b9a
paper_authors.avatarUrl: /avatars/9156dc406ed3f9ee62b73657ac20f5ed.svg
paper_authors.fullname: Yann LeCun
paper_authors.type: user
paper_authors.user: ylecun
paper_authors._id: 655ec78fa3e81145767e66a9
paper_authors.name: Thomas Scialom
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-23T10:52:47.274Z
paper_authors._id: 60c0e0c06c0791283058f0fb
paper_authors.avatarUrl: /avatars/84c0fe390a171e68b79a8274405ebf06.svg
paper_authors.fullname: Scialom
paper_authors.type: user
paper_authors.user: ThomasNLG
paper_title: GAIA: a benchmark for General AI Assistants
paper_summary: We introduce GAIA, a benchmark for General AI Assistants that, if solved,
  would represent a milestone in AI research. GAIA proposes real-world questions
  that require a set of fundamental abilities such as reasoning, multi-modality
  handling, web browsing, and generally tool-use proficiency. GAIA questions are
  conceptually simple for humans yet challenging for most advanced AIs: we show
  that human respondents obtain 92\% vs. 15\% for GPT-4 equipped with plugins.
  This notable performance disparity contrasts with the recent trend of LLMs
  outperforming humans on tasks requiring professional skills in e.g. law or
  chemistry. GAIA's philosophy departs from the current trend in AI benchmarks
  suggesting to target tasks that are ever more difficult for humans. We posit
  that the advent of Artificial General Intelligence (AGI) hinges on a system's
  capability to exhibit similar robustness as the average human does on such
  questions. Using GAIA's methodology, we devise 466 questions and their answer.
  We release our questions while retaining answers to 300 of them to power a
  leader-board available at https://huggingface.co/gaia-benchmark.
paper_ai_keywords: General AI Assistants (GAIA) in-the-loop multi-modality handling real-world questions reasoning rapid browsing general-purpose proficiency classification performance Large Language Models (LLMs) Artificial General Intelligence (AGI) benchmark methodology