title: Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large
    Language Models
summary: Recent advancements in large language models (LLMs) have led to significant
  breakthroughs in mathematical reasoning capabilities. However, existing
  benchmarks like GSM8K or MATH are now being solved with high accuracy (e.g.,
  OpenAI o1 achieves 94.8% on MATH dataset), indicating their inadequacy for
  truly challenging these models. To bridge this gap, we propose a comprehensive
  and challenging benchmark specifically designed to assess LLMs' mathematical
  reasoning at the Olympiad level. Unlike existing Olympiad-related benchmarks,
  our dataset focuses exclusively on mathematics and comprises a vast collection
  of 4428 competition-level problems with rigorous human annotation. These
  problems are meticulously categorized into over 33 sub-domains and span more
  than 10 distinct difficulty levels, enabling a holistic assessment of model
  performance in Olympiad-mathematical reasoning. Furthermore, we conducted an
  in-depth analysis based on this benchmark. Our experimental results show that
  even the most advanced models, OpenAI o1-mini and OpenAI o1-preview, struggle
  with highly challenging Olympiad-level problems, with 60.54% and 52.55%
  accuracy, highlighting significant challenges in Olympiad-level mathematical
  reasoning.
paper_authors._id: 6709313569a07aa53e3f5e88
paper_authors.name: Bofei Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-15T13:58:13.843Z
paper_authors._id: 65ae21adabf6d1ccb795e9a4
paper_authors.avatarUrl: /avatars/b5dced62c6a3564095a8fa0959bc06cb.svg
paper_authors.fullname: Bofei Gao
paper_authors.type: user
paper_authors.user: KbsdJames
paper_authors._id: 6709313569a07aa53e3f5e89
paper_authors.name: Feifan Song
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-15T14:00:23.150Z
paper_authors._id: 6447ca6ca478b20f1755b294
paper_authors.avatarUrl: /avatars/5049856b5ed1b74533fff902e14b4c7c.svg
paper_authors.fullname: Feifan Song
paper_authors.type: user
paper_authors.user: songff
paper_authors._id: 6709313569a07aa53e3f5e8a
paper_authors.name: Zhe Yang
paper_authors._id: 6709313569a07aa53e3f5e8b
paper_authors.name: Zefan Cai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-15T14:01:36.446Z
paper_authors._id: 64b15284372d4340772a3dca
paper_authors.avatarUrl: /avatars/417d5f1bc1bcb5e4d5de6169673c2cf7.svg
paper_authors.fullname: Zefan Cai
paper_authors.type: user
paper_authors.user: ZefanCai
paper_authors._id: 6709313569a07aa53e3f5e8c
paper_authors.name: Yibo Miao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-15T14:01:42.392Z
paper_authors._id: 64a139c098fad0c8a5a627a4
paper_authors.avatarUrl: /avatars/6eb508abd827d4a4f5abb6b24155b22d.svg
paper_authors.fullname: Yibo Miao
paper_authors.type: user
paper_authors.user: instro
paper_authors._id: 6709313569a07aa53e3f5e8d
paper_authors.name: Qingxiu Dong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-15T14:01:48.414Z
paper_authors._id: 670740744341dcee459fb990
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/66UkZvrAk7fQr5YCylEFk.png
paper_authors.fullname: Qingxiu Dong
paper_authors.type: user
paper_authors.user: Rsy24
paper_authors._id: 6709313569a07aa53e3f5e8e
paper_authors.name: Lei Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-15T09:11:16.735Z
paper_authors._id: 6038d6d0612f5eef3cc05ea9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6038d6d0612f5eef3cc05ea9/ryhvAX5djQpD5OrIlZQ1f.jpeg
paper_authors.fullname: Lei Li
paper_authors.type: user
paper_authors.user: tobiaslee
paper_authors._id: 6709313569a07aa53e3f5e8f
paper_authors.name: Chenghao Ma
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-15T09:11:18.805Z
paper_authors._id: 66cd868be2ed0c6657eefeb7
paper_authors.avatarUrl: /avatars/72c33b6c17a4f69c67c76cdde15f54eb.svg
paper_authors.fullname: mch
paper_authors.type: user
paper_authors.user: mch0115
paper_authors._id: 6709313569a07aa53e3f5e90
paper_authors.name: Liang Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-15T14:02:05.684Z
paper_authors._id: 658c481dd1c8b106727a8b73
paper_authors.avatarUrl: /avatars/d34a7a62c3a524e5fdd2d5994348db58.svg
paper_authors.fullname: Liang Chen
paper_authors.type: user
paper_authors.user: liangchen-ms
paper_authors._id: 6709313569a07aa53e3f5e91
paper_authors.name: Runxin Xu
paper_authors._id: 6709313569a07aa53e3f5e92
paper_authors.name: Zhengyang Tang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-15T14:02:20.963Z
paper_authors._id: 64912976b95c3f0a1e6233cb
paper_authors.avatarUrl: /avatars/c0615f8c6606073faffb419757d4e667.svg
paper_authors.fullname: Zhengyang Tang
paper_authors.type: user
paper_authors.user: tangzhy
paper_authors._id: 6709313569a07aa53e3f5e93
paper_authors.name: Benyou Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-15T14:02:30.698Z
paper_authors._id: 637c6703ca8542a0ba900ccb
paper_authors.avatarUrl: /avatars/288ed63a1efa566c3f01e850c6ba5dd5.svg
paper_authors.fullname: Wang
paper_authors.type: user
paper_authors.user: Benyou
paper_authors._id: 6709313569a07aa53e3f5e94
paper_authors.name: Daoguang Zan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-15T14:02:36.972Z
paper_authors._id: 61527edf8b55dbdae72874fa
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61527edf8b55dbdae72874fa/ZGWSBf_KSrDof6WyMoDMU.jpeg
paper_authors.fullname: Daoguang Zan
paper_authors.type: user
paper_authors.user: Daoguang
paper_authors._id: 6709313569a07aa53e3f5e95
paper_authors.name: Shanghaoran Quan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-25T09:31:19.169Z
paper_authors._id: 64b9954845ce8d7ad607c14d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64b9954845ce8d7ad607c14d/LZ9yeTOz4J_YKrnGkcmnL.jpeg
paper_authors.fullname: Shanghaoran Quan
paper_authors.type: user
paper_authors.user: quanshr
paper_authors._id: 6709313569a07aa53e3f5e96
paper_authors.name: Ge Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-15T14:02:52.819Z
paper_authors._id: 638efcf4c67af472d316d424
paper_authors.avatarUrl: /avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
paper_authors.fullname: Ge Zhang
paper_authors.type: user
paper_authors.user: zhangysk
paper_authors._id: 6709313569a07aa53e3f5e97
paper_authors.name: Lei Sha
paper_authors._id: 6709313569a07aa53e3f5e98
paper_authors.name: Yichang Zhang
paper_authors._id: 6709313569a07aa53e3f5e99
paper_authors.name: Xuancheng Ren
paper_authors._id: 6709313569a07aa53e3f5e9a
paper_authors.name: Tianyu Liu
paper_authors._id: 6709313569a07aa53e3f5e9b
paper_authors.name: Baobao Chang
paper_title: Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large
    Language Models
paper_summary: Recent advancements in large language models (LLMs) have led to significant
  breakthroughs in mathematical reasoning capabilities. However, existing
  benchmarks like GSM8K or MATH are now being solved with high accuracy (e.g.,
  OpenAI o1 achieves 94.8% on MATH dataset), indicating their inadequacy for
  truly challenging these models. To bridge this gap, we propose a comprehensive
  and challenging benchmark specifically designed to assess LLMs' mathematical
  reasoning at the Olympiad level. Unlike existing Olympiad-related benchmarks,
  our dataset focuses exclusively on mathematics and comprises a vast collection
  of 4428 competition-level problems with rigorous human annotation. These
  problems are meticulously categorized into over 33 sub-domains and span more
  than 10 distinct difficulty levels, enabling a holistic assessment of model
  performance in Olympiad-mathematical reasoning. Furthermore, we conducted an
  in-depth analysis based on this benchmark. Our experimental results show that
  even the most advanced models, OpenAI o1-mini and OpenAI o1-preview, struggle
  with highly challenging Olympiad-level problems, with 60.54% and 52.55%
  accuracy, highlighting significant challenges in Olympiad-level mathematical
  reasoning.
paper_ai_keywords: large language models (LLMs) mathematical reasoning capabilities benchmarking GSM8K MATH OpenAI o1 accuracy dataset Olympiad-level benchmark semantic annotations sub-domains difficulty levels holistic assessment model performance Olympiad-mathematical reasoning OpenAI o1-mini OpenAI o1-preview