title: OlympicArena Medal Ranks: Who Is the Most Intelligent AI So Far?
summary: In this report, we pose the following question: Who is the most intelligent
  AI model to date, as measured by the OlympicArena (an Olympic-level,
  multi-discipline, multi-modal benchmark for superintelligent AI)? We
  specifically focus on the most recently released models: Claude-3.5-Sonnet,
  Gemini-1.5-Pro, and GPT-4o. For the first time, we propose using an Olympic
  medal Table approach to rank AI models based on their comprehensive performance
  across various disciplines. Empirical results reveal: (1) Claude-3.5-Sonnet
  shows highly competitive overall performance over GPT-4o, even surpassing
  GPT-4o on a few subjects (i.e., Physics, Chemistry, and Biology). (2)
  Gemini-1.5-Pro and GPT-4V are ranked consecutively just behind GPT-4o and
  Claude-3.5-Sonnet, but with a clear performance gap between them. (3) The
  performance of AI models from the open-source community significantly lags
  behind these proprietary models. (4) The performance of these models on this
  benchmark has been less than satisfactory, indicating that we still have a long
  way to go before achieving superintelligence. We remain committed to
  continuously tracking and evaluating the performance of the latest powerful
  models on this benchmark (available at
  https://github.com/GAIR-NLP/OlympicArena).
paper_authors._id: 667a721deb164e5584d41568
paper_authors.name: Zhen Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-25T13:44:11.851Z
paper_authors._id: 643581a4f3b08e267d990499
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/643581a4f3b08e267d990499/KRhB-48W4IPuB0bX16Ahj.png
paper_authors.fullname: Zhen Huang
paper_authors.type: user
paper_authors.user: ZhenHuang
paper_authors._id: 667a721deb164e5584d41569
paper_authors.name: Zengzhi Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-25T07:33:34.375Z
paper_authors._id: 62cbeb2d72dfd24b86bdf977
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62cbeb2d72dfd24b86bdf977/UcGYYSBNrCvPM5K9v-sro.png
paper_authors.fullname: Zengzhi Wang
paper_authors.type: user
paper_authors.user: SinclairWang
paper_authors._id: 667a721deb164e5584d4156a
paper_authors.name: Shijie Xia
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-25T13:44:28.419Z
paper_authors._id: 65900d4ff5a209eeac08b463
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65900d4ff5a209eeac08b463/PJNNBRJIk1qR24oaRLTex.jpeg
paper_authors.fullname: shijie xia
paper_authors.type: user
paper_authors.user: seven-cat
paper_authors._id: 667a721deb164e5584d4156b
paper_authors.name: Pengfei Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-25T13:44:40.774Z
paper_authors._id: 6144a0c4ff1146bbd84d9865
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1661715958139-6144a0c4ff1146bbd84d9865.png
paper_authors.fullname: Pengfei Liu
paper_authors.type: user
paper_authors.user: Pengfei
paper_title: OlympicArena Medal Ranks: Who Is the Most Intelligent AI So Far?
paper_summary: In this report, we pose the following question: Who is the most intelligent
  AI model to date, as measured by the OlympicArena (an Olympic-level,
  multi-discipline, multi-modal benchmark for superintelligent AI)? We
  specifically focus on the most recently released models: Claude-3.5-Sonnet,
  Gemini-1.5-Pro, and GPT-4o. For the first time, we propose using an Olympic
  medal Table approach to rank AI models based on their comprehensive performance
  across various disciplines. Empirical results reveal: (1) Claude-3.5-Sonnet
  shows highly competitive overall performance over GPT-4o, even surpassing
  GPT-4o on a few subjects (i.e., Physics, Chemistry, and Biology). (2)
  Gemini-1.5-Pro and GPT-4V are ranked consecutively just behind GPT-4o and
  Claude-3.5-Sonnet, but with a clear performance gap between them. (3) The
  performance of AI models from the open-source community significantly lags
  behind these proprietary models. (4) The performance of these models on this
  benchmark has been less than satisfactory, indicating that we still have a long
  way to go before achieving superintelligence. We remain committed to
  continuously tracking and evaluating the performance of the latest powerful
  models on this benchmark (available at
  https://github.com/GAIR-NLP/OlympicArena).
paper_ai_summary: Evaluation of AI models Claude-3.5-Sonnet, Gemini-1.5-Pro, and GPT-4o on the OlympicArena benchmark demonstrates Claude-3.5-Sonnet's competitive overall performance and leads to insights about the current state of AI superintelligence.
paper_ai_keywords: OlympicArena multi-discipline multi-modal benchmark superintelligent AI Olympic medal Table Claude-3.5-Sonnet Gemini-1.5-Pro GPT-4o GPT-4V open-source community