OlympicArena Medal Ranks: Who Is the Most Intelligent AI So Far?
In this report, we pose the following question: Who is the most intelligent
AI model to date, as measured by the OlympicArena (an Olympic-level,
multi-discipline, multi-modal benchmark for superintelligent AI)? We
specifically focus on the most recently released models: Claude-3.5-Sonnet,
Gemini-1.5-Pro, and GPT-4o. For the first time, we propose using an Olympic
medal Table approach to rank AI models based on their comprehensive performance
across various disciplines. Empirical results reveal: (1) Claude-3.5-Sonnet
shows highly competitive overall performance over GPT-4o, even surpassing
GPT-4o on a few subjects (i.e., Physics, Chemistry, and Biology). (2)
Gemini-1.5-Pro and GPT-4V are ranked consecutively just behind GPT-4o and
Claude-3.5-Sonnet, but with a clear performance gap between them. (3) The
performance of AI models from the open-source community significantly lags
behind these proprietary models. (4) The performance of these models on this
benchmark has been less than satisfactory, indicating that we still have a long
way to go before achieving superintelligence. We remain committed to
continuously tracking and evaluating the performance of the latest powerful
models on this benchmark (available at
https://github.com/GAIR-NLP/OlympicArena).
667a721deb164e5584d41568
Zhen Huang
admin_assigned
2024-06-25T13:44:11.851Z
643581a4f3b08e267d990499
https://cdn-avatars.huggingface.co/v1/production/uploads/643581a4f3b08e267d990499/KRhB-48W4IPuB0bX16Ahj.png
Zhen Huang
user
ZhenHuang
667a721deb164e5584d41569
Zengzhi Wang
claimed_verified
2024-06-25T07:33:34.375Z
62cbeb2d72dfd24b86bdf977
https://cdn-avatars.huggingface.co/v1/production/uploads/62cbeb2d72dfd24b86bdf977/UcGYYSBNrCvPM5K9v-sro.png
Zengzhi Wang
user
SinclairWang
667a721deb164e5584d4156a
Shijie Xia
admin_assigned
2024-06-25T13:44:28.419Z
65900d4ff5a209eeac08b463
https://cdn-avatars.huggingface.co/v1/production/uploads/65900d4ff5a209eeac08b463/PJNNBRJIk1qR24oaRLTex.jpeg
shijie xia
user
seven-cat
667a721deb164e5584d4156b
Pengfei Liu
admin_assigned
2024-06-25T13:44:40.774Z
6144a0c4ff1146bbd84d9865
https://cdn-avatars.huggingface.co/v1/production/uploads/1661715958139-6144a0c4ff1146bbd84d9865.png
Pengfei Liu
user
Pengfei
OlympicArena Medal Ranks: Who Is the Most Intelligent AI So Far?
In this report, we pose the following question: Who is the most intelligent
AI model to date, as measured by the OlympicArena (an Olympic-level,
multi-discipline, multi-modal benchmark for superintelligent AI)? We
specifically focus on the most recently released models: Claude-3.5-Sonnet,
Gemini-1.5-Pro, and GPT-4o. For the first time, we propose using an Olympic
medal Table approach to rank AI models based on their comprehensive performance
across various disciplines. Empirical results reveal: (1) Claude-3.5-Sonnet
shows highly competitive overall performance over GPT-4o, even surpassing
GPT-4o on a few subjects (i.e., Physics, Chemistry, and Biology). (2)
Gemini-1.5-Pro and GPT-4V are ranked consecutively just behind GPT-4o and
Claude-3.5-Sonnet, but with a clear performance gap between them. (3) The
performance of AI models from the open-source community significantly lags
behind these proprietary models. (4) The performance of these models on this
benchmark has been less than satisfactory, indicating that we still have a long
way to go before achieving superintelligence. We remain committed to
continuously tracking and evaluating the performance of the latest powerful
models on this benchmark (available at
https://github.com/GAIR-NLP/OlympicArena).
Evaluation of AI models Claude-3.5-Sonnet, Gemini-1.5-Pro, and GPT-4o on the OlympicArena benchmark demonstrates Claude-3.5-Sonnet's competitive overall performance and leads to insights about the current state of AI superintelligence.
OlympicArena multi-discipline multi-modal benchmark superintelligent AI Olympic medal Table Claude-3.5-Sonnet Gemini-1.5-Pro GPT-4o GPT-4V open-source community