Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large
  Language Models
Recent advancements in large language models (LLMs) have led to significant
breakthroughs in mathematical reasoning capabilities. However, existing
benchmarks like GSM8K or MATH are now being solved with high accuracy (e.g.,
OpenAI o1 achieves 94.8% on MATH dataset), indicating their inadequacy for
truly challenging these models. To bridge this gap, we propose a comprehensive
and challenging benchmark specifically designed to assess LLMs' mathematical
reasoning at the Olympiad level. Unlike existing Olympiad-related benchmarks,
our dataset focuses exclusively on mathematics and comprises a vast collection
of 4428 competition-level problems with rigorous human annotation. These
problems are meticulously categorized into over 33 sub-domains and span more
than 10 distinct difficulty levels, enabling a holistic assessment of model
performance in Olympiad-mathematical reasoning. Furthermore, we conducted an
in-depth analysis based on this benchmark. Our experimental results show that
even the most advanced models, OpenAI o1-mini and OpenAI o1-preview, struggle
with highly challenging Olympiad-level problems, with 60.54% and 52.55%
accuracy, highlighting significant challenges in Olympiad-level mathematical
reasoning.
6709313569a07aa53e3f5e88
Bofei Gao
admin_assigned
2024-10-15T13:58:13.843Z
65ae21adabf6d1ccb795e9a4
/avatars/b5dced62c6a3564095a8fa0959bc06cb.svg
Bofei Gao
user
KbsdJames
6709313569a07aa53e3f5e89
Feifan Song
admin_assigned
2024-10-15T14:00:23.150Z
6447ca6ca478b20f1755b294
/avatars/5049856b5ed1b74533fff902e14b4c7c.svg
Feifan Song
user
songff
6709313569a07aa53e3f5e8a
Zhe Yang
6709313569a07aa53e3f5e8b
Zefan Cai
admin_assigned
2024-10-15T14:01:36.446Z
64b15284372d4340772a3dca
/avatars/417d5f1bc1bcb5e4d5de6169673c2cf7.svg
Zefan Cai
user
ZefanCai
6709313569a07aa53e3f5e8c
Yibo Miao
admin_assigned
2024-10-15T14:01:42.392Z
64a139c098fad0c8a5a627a4
https://cdn-avatars.huggingface.co/v1/production/uploads/64a139c098fad0c8a5a627a4/MgWortQS64cJOZ3pL4WyH.jpeg
Yibo Miao
user
instro
6709313569a07aa53e3f5e8d
Qingxiu Dong
admin_assigned
2024-10-15T14:01:48.414Z
670740744341dcee459fb990
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/66UkZvrAk7fQr5YCylEFk.png
Qingxiu Dong
user
Rsy24
6709313569a07aa53e3f5e8e
Lei Li
claimed_verified
2024-10-15T09:11:16.735Z
6038d6d0612f5eef3cc05ea9
https://cdn-avatars.huggingface.co/v1/production/uploads/6038d6d0612f5eef3cc05ea9/ryhvAX5djQpD5OrIlZQ1f.jpeg
Lei Li
user
tobiaslee
6709313569a07aa53e3f5e8f
Chenghao Ma
claimed_verified
2024-10-15T09:11:18.805Z
66cd868be2ed0c6657eefeb7
/avatars/72c33b6c17a4f69c67c76cdde15f54eb.svg
mch
user
mch0115
6709313569a07aa53e3f5e90
Liang Chen
admin_assigned
2024-10-15T14:02:05.684Z
658c481dd1c8b106727a8b73
/avatars/d34a7a62c3a524e5fdd2d5994348db58.svg
Liang Chen
user
liangchen-ms
6709313569a07aa53e3f5e91
Runxin Xu
6709313569a07aa53e3f5e92
Zhengyang Tang
admin_assigned
2024-10-15T14:02:20.963Z
64912976b95c3f0a1e6233cb
/avatars/c0615f8c6606073faffb419757d4e667.svg
Zhengyang Tang
user
tangzhy
6709313569a07aa53e3f5e93
Benyou Wang
admin_assigned
2024-10-15T14:02:30.698Z
637c6703ca8542a0ba900ccb
/avatars/288ed63a1efa566c3f01e850c6ba5dd5.svg
Wang
user
Benyou
6709313569a07aa53e3f5e94
Daoguang Zan
admin_assigned
2024-10-15T14:02:36.972Z
61527edf8b55dbdae72874fa
https://cdn-avatars.huggingface.co/v1/production/uploads/61527edf8b55dbdae72874fa/ZGWSBf_KSrDof6WyMoDMU.jpeg
Daoguang Zan
user
Daoguang
6709313569a07aa53e3f5e95
Shanghaoran Quan
claimed_verified
2024-10-25T09:31:19.169Z
64b9954845ce8d7ad607c14d
https://cdn-avatars.huggingface.co/v1/production/uploads/64b9954845ce8d7ad607c14d/LZ9yeTOz4J_YKrnGkcmnL.jpeg
Shanghaoran Quan
user
quanshr
6709313569a07aa53e3f5e96
Ge Zhang
admin_assigned
2024-10-15T14:02:52.819Z
638efcf4c67af472d316d424
/avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
Ge Zhang
user
zhangysk
6709313569a07aa53e3f5e97
Lei Sha
6709313569a07aa53e3f5e98
Yichang Zhang
6709313569a07aa53e3f5e99
Xuancheng Ren
6709313569a07aa53e3f5e9a
Tianyu Liu
6709313569a07aa53e3f5e9b
Baobao Chang
Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large
  Language Models
Recent advancements in large language models (LLMs) have led to significant
breakthroughs in mathematical reasoning capabilities. However, existing
benchmarks like GSM8K or MATH are now being solved with high accuracy (e.g.,
OpenAI o1 achieves 94.8% on MATH dataset), indicating their inadequacy for
truly challenging these models. To bridge this gap, we propose a comprehensive
and challenging benchmark specifically designed to assess LLMs' mathematical
reasoning at the Olympiad level. Unlike existing Olympiad-related benchmarks,
our dataset focuses exclusively on mathematics and comprises a vast collection
of 4428 competition-level problems with rigorous human annotation. These
problems are meticulously categorized into over 33 sub-domains and span more
than 10 distinct difficulty levels, enabling a holistic assessment of model
performance in Olympiad-mathematical reasoning. Furthermore, we conducted an
in-depth analysis based on this benchmark. Our experimental results show that
even the most advanced models, OpenAI o1-mini and OpenAI o1-preview, struggle
with highly challenging Olympiad-level problems, with 60.54% and 52.55%
accuracy, highlighting significant challenges in Olympiad-level mathematical
reasoning.
A new benchmark evaluates LLMs on Olympiad-level mathematics, revealing challenges even for advanced models.
large language models LLMs mathematical reasoning GSM8K MATH OpenAI o1 benchmark competition-level problems human annotation mathematical reasoning at Olympiad level sub-domains difficulty levels OpenAI o1-mini OpenAI o1-preview Olympiad-level mathematical reasoning