title: OlympicArena: Benchmarking Multi-discipline Cognitive Reasoning for
    Superintelligent AI
summary: The evolution of Artificial Intelligence (AI) has been significantly
  accelerated by advancements in Large Language Models (LLMs) and Large
  Multimodal Models (LMMs), gradually showcasing potential cognitive reasoning
  abilities in problem-solving and scientific discovery (i.e., AI4Science) once
  exclusive to human intellect. To comprehensively evaluate current models'
  performance in cognitive reasoning abilities, we introduce OlympicArena, which
  includes 11,163 bilingual problems across both text-only and interleaved
  text-image modalities. These challenges encompass a wide range of disciplines
  spanning seven fields and 62 international Olympic competitions, rigorously
  examined for data leakage. We argue that the challenges in Olympic competition
  problems are ideal for evaluating AI's cognitive reasoning due to their
  complexity and interdisciplinary nature, which are essential for tackling
  complex scientific challenges and facilitating discoveries. Beyond evaluating
  performance across various disciplines using answer-only criteria, we conduct
  detailed experiments and analyses from multiple perspectives. We delve into the
  models' cognitive reasoning abilities, their performance across different
  modalities, and their outcomes in process-level evaluations, which are vital
  for tasks requiring complex reasoning with lengthy solutions. Our extensive
  evaluations reveal that even advanced models like GPT-4o only achieve a 39.97%
  overall accuracy, illustrating current AI limitations in complex reasoning and
  multimodal integration. Through the OlympicArena, we aim to advance AI towards
  superintelligence, equipping it to address more complex challenges in science
  and beyond. We also provide a comprehensive set of resources to support AI
  research, including a benchmark dataset, an open-source annotation platform, a
  detailed evaluation tool, and a leaderboard with automatic submission features.
paper_authors._id: 66724205c12108af6ccd1d3b
paper_authors.name: Zhen Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-19T20:49:48.170Z
paper_authors._id: 6416af7a58a690df1040f422
paper_authors.avatarUrl: /avatars/224d4015382063e713a99f96bf332722.svg
paper_authors.fullname: Zhen Huang
paper_authors.type: user
paper_authors.user: Benjamin02
paper_authors._id: 66724205c12108af6ccd1d3c
paper_authors.name: Zengzhi Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-19T21:00:36.085Z
paper_authors._id: 62cbeb2d72dfd24b86bdf977
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62cbeb2d72dfd24b86bdf977/UcGYYSBNrCvPM5K9v-sro.png
paper_authors.fullname: Zengzhi Wang
paper_authors.type: user
paper_authors.user: SinclairWang
paper_authors._id: 66724205c12108af6ccd1d3d
paper_authors.name: Shijie Xia
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-19T21:00:41.740Z
paper_authors._id: 65900d4ff5a209eeac08b463
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65900d4ff5a209eeac08b463/PJNNBRJIk1qR24oaRLTex.jpeg
paper_authors.fullname: shijie xia
paper_authors.type: user
paper_authors.user: seven-cat
paper_authors._id: 66724205c12108af6ccd1d3e
paper_authors.name: Xuefeng Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-19T21:00:59.064Z
paper_authors._id: 64e3562342d8e2c1c69f64b3
paper_authors.avatarUrl: /avatars/1ccf22d60deec213bbe069d30811efbe.svg
paper_authors.fullname: xuefengli
paper_authors.type: user
paper_authors.user: xuefengli
paper_authors._id: 66724205c12108af6ccd1d3f
paper_authors.name: Haoyang Zou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-19T21:01:05.197Z
paper_authors._id: 65011b18af1e4021439af578
paper_authors.avatarUrl: /avatars/ae1632c800ceeb43aa223e432f70bc6d.svg
paper_authors.fullname: Haoyang Zou
paper_authors.type: user
paper_authors.user: alanyoung058
paper_authors._id: 66724205c12108af6ccd1d40
paper_authors.name: Ruijie Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-19T21:01:11.785Z
paper_authors._id: 6537cdda25f780fed2375a7b
paper_authors.avatarUrl: /avatars/6309fbcbb97cd63957ed7429bbe311d0.svg
paper_authors.fullname: Ruijie Xu
paper_authors.type: user
paper_authors.user: GardenStrayer
paper_authors._id: 66724205c12108af6ccd1d41
paper_authors.name: Run-Ze Fan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-19T07:38:50.815Z
paper_authors._id: 616bfc2b40e2f69baa1c7add
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/616bfc2b40e2f69baa1c7add/Os7_qgMei-2lRVelrOG7B.jpeg
paper_authors.fullname: Run-Ze Fan
paper_authors.type: user
paper_authors.user: Vfrz
paper_authors._id: 66724205c12108af6ccd1d42
paper_authors.name: Lyumanshan Ye
paper_authors._id: 66724205c12108af6ccd1d43
paper_authors.name: Ethan Chern
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-19T21:01:22.700Z
paper_authors._id: 64bb5f9d8e051085bace4d1e
paper_authors.avatarUrl: /avatars/15ccbb78c6131dfe46b7a9d8e7d1a31f.svg
paper_authors.fullname: Ethan Chern
paper_authors.type: user
paper_authors.user: ethanchern
paper_authors._id: 66724205c12108af6ccd1d44
paper_authors.name: Yixin Ye
paper_authors._id: 66724205c12108af6ccd1d45
paper_authors.name: Yikai Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-19T21:01:51.271Z
paper_authors._id: 63e8b792ca4fc7d30de6975b
paper_authors.avatarUrl: /avatars/57237f54d61d479df15209497a3f531e.svg
paper_authors.fullname: Yikai Zhang
paper_authors.type: user
paper_authors.user: Arist12
paper_authors._id: 66724205c12108af6ccd1d46
paper_authors.name: Yuqing Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-26T08:50:10.437Z
paper_authors._id: 65c0de12efbb14b39c97f78e
paper_authors.avatarUrl: /avatars/18485f79427a35bd9e19f71b67c88dce.svg
paper_authors.fullname: Yuqing Yang
paper_authors.type: user
paper_authors.user: ayyyq
paper_authors._id: 66724205c12108af6ccd1d47
paper_authors.name: Ting Wu
paper_authors._id: 66724205c12108af6ccd1d48
paper_authors.name: Binjie Wang
paper_authors._id: 66724205c12108af6ccd1d49
paper_authors.name: Shichao Sun
paper_authors._id: 66724205c12108af6ccd1d4a
paper_authors.name: Yang Xiao
paper_authors._id: 66724205c12108af6ccd1d4b
paper_authors.name: Yiyuan Li
paper_authors._id: 66724205c12108af6ccd1d4c
paper_authors.name: Fan Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-19T20:20:51.598Z
paper_authors._id: 628f6e5ab90dde28ef57d293
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/628f6e5ab90dde28ef57d293/AxNzR2nvrND6Rf3RPkYMk.jpeg
paper_authors.fullname: Fan Zhou
paper_authors.type: user
paper_authors.user: koalazf99
paper_authors._id: 66724205c12108af6ccd1d4d
paper_authors.name: Steffi Chern
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-19T21:03:14.603Z
paper_authors._id: 64b370fe6d953e7c75ede314
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64b370fe6d953e7c75ede314/RdP2q3hGXWE4E2zfSv0KU.png
paper_authors.fullname: Steffi Chern
paper_authors.type: user
paper_authors.user: steffichern
paper_authors._id: 66724205c12108af6ccd1d4e
paper_authors.name: Yiwei Qin
paper_authors._id: 66724205c12108af6ccd1d4f
paper_authors.name: Yan Ma
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-26T08:50:13.002Z
paper_authors._id: 633fc70529b5a95f6e15a6b7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/633fc70529b5a95f6e15a6b7/Fzh7wWuqU-fBbzdupOUtF.jpeg
paper_authors.fullname: Yan Ma
paper_authors.type: user
paper_authors.user: ManTle
paper_authors._id: 66724205c12108af6ccd1d50
paper_authors.name: Jiadi Su
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-19T21:03:35.224Z
paper_authors._id: 65437c82d3db5d5336af93ff
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65437c82d3db5d5336af93ff/YDfpywk15IigQwcAoMu5n.jpeg
paper_authors.fullname: Jiadi Su
paper_authors.type: user
paper_authors.user: JoyBoy-Su
paper_authors._id: 66724205c12108af6ccd1d51
paper_authors.name: Yixiu Liu
paper_authors._id: 66724205c12108af6ccd1d52
paper_authors.name: Yuxiang Zheng
paper_authors._id: 66724205c12108af6ccd1d53
paper_authors.name: Shaoting Zhang
paper_authors._id: 66724205c12108af6ccd1d54
paper_authors.name: Dahua Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-19T21:03:49.944Z
paper_authors._id: 636317ed80c1a705a6eff396
paper_authors.avatarUrl: /avatars/3db090e101b916d9256d0d3e043db71d.svg
paper_authors.fullname: Dahua Lin
paper_authors.type: user
paper_authors.user: lindahua
paper_authors._id: 66724205c12108af6ccd1d55
paper_authors.name: Yu Qiao
paper_authors._id: 66724205c12108af6ccd1d56
paper_authors.name: Pengfei Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-19T21:03:56.925Z
paper_authors._id: 6144a0c4ff1146bbd84d9865
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1661715958139-6144a0c4ff1146bbd84d9865.png
paper_authors.fullname: Pengfei Liu
paper_authors.type: user
paper_authors.user: Pengfei
paper_title: OlympicArena: Benchmarking Multi-discipline Cognitive Reasoning for
    Superintelligent AI
paper_summary: The evolution of Artificial Intelligence (AI) has been significantly
  accelerated by advancements in Large Language Models (LLMs) and Large
  Multimodal Models (LMMs), gradually showcasing potential cognitive reasoning
  abilities in problem-solving and scientific discovery (i.e., AI4Science) once
  exclusive to human intellect. To comprehensively evaluate current models'
  performance in cognitive reasoning abilities, we introduce OlympicArena, which
  includes 11,163 bilingual problems across both text-only and interleaved
  text-image modalities. These challenges encompass a wide range of disciplines
  spanning seven fields and 62 international Olympic competitions, rigorously
  examined for data leakage. We argue that the challenges in Olympic competition
  problems are ideal for evaluating AI's cognitive reasoning due to their
  complexity and interdisciplinary nature, which are essential for tackling
  complex scientific challenges and facilitating discoveries. Beyond evaluating
  performance across various disciplines using answer-only criteria, we conduct
  detailed experiments and analyses from multiple perspectives. We delve into the
  models' cognitive reasoning abilities, their performance across different
  modalities, and their outcomes in process-level evaluations, which are vital
  for tasks requiring complex reasoning with lengthy solutions. Our extensive
  evaluations reveal that even advanced models like GPT-4o only achieve a 39.97%
  overall accuracy, illustrating current AI limitations in complex reasoning and
  multimodal integration. Through the OlympicArena, we aim to advance AI towards
  superintelligence, equipping it to address more complex challenges in science
  and beyond. We also provide a comprehensive set of resources to support AI
  research, including a benchmark dataset, an open-source annotation platform, a
  detailed evaluation tool, and a leaderboard with automatic submission features.
paper_ai_summary: OlympicArena, a benchmark for evaluating AI's cognitive reasoning across disciplines, demonstrates current models' limitations and provides resources to advance AI towards superintelligence.
paper_ai_keywords: Large Language Models LLMs Large Multimodal Models LMMs AI4Science cognitive reasoning OlympicArena bilingual problems text-only interleaved text-image cognitive reasoning abilities complexity interdisciplinary nature data leakage advanced models GPT-4o multimodal integration