title: Challenging the Boundaries of Reasoning: An Olympiad-Level Math
    Benchmark for Large Language Models
summary: In recent years, the rapid development of large reasoning models has resulted
  in the saturation of existing benchmarks for evaluating mathematical reasoning,
  highlighting the urgent need for more challenging and rigorous evaluation
  frameworks. To address this gap, we introduce OlymMATH, a novel Olympiad-level
  mathematical benchmark, designed to rigorously test the complex reasoning
  capabilities of LLMs. OlymMATH features 200 meticulously curated problems, each
  manually verified and available in parallel English and Chinese versions. The
  problems are systematically organized into two distinct difficulty tiers: (1)
  AIME-level problems (easy) that establish a baseline for mathematical reasoning
  assessment, and (2) significantly more challenging problems (hard) designed to
  push the boundaries of current state-of-the-art models. In our benchmark, these
  problems span four core mathematical fields, each including a verifiable
  numerical solution to enable objective, rule-based evaluation. Empirical
  results underscore the significant challenge presented by OlymMATH, with
  state-of-the-art models including DeepSeek-R1 and OpenAI's o3-mini
  demonstrating notably limited accuracy on the hard subset. Furthermore, the
  benchmark facilitates comprehensive bilingual assessment of mathematical
  reasoning abilities-a critical dimension that remains largely unaddressed in
  mainstream mathematical reasoning benchmarks. We release the OlymMATH benchmark
  at the STILL project: https://github.com/RUCAIBox/Slow_Thinking_with_LLMs.
paper_authors._id: 67e5f4ad147ee85622ad0df1
paper_authors.name: Haoxiang Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-28T08:46:46.418Z
paper_authors._id: 65df408822d66a997b4d5f6e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65df408822d66a997b4d5f6e/poROuCSvB39NZSiLzxLZf.jpeg
paper_authors.fullname: Haoxiang Sun
paper_authors.type: user
paper_authors.user: CoderBak
paper_authors._id: 67e5f4ad147ee85622ad0df2
paper_authors.name: Yingqian Min
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-28T08:38:03.334Z
paper_authors._id: 6703ac76ea890f0ca5b225eb
paper_authors.avatarUrl: /avatars/5f56c49a1940143d47dd484782a4abbf.svg
paper_authors.fullname: Yingqian Min
paper_authors.type: user
paper_authors.user: EliverQ
paper_authors._id: 67e5f4ad147ee85622ad0df3
paper_authors.name: Zhipeng Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-28T08:38:01.177Z
paper_authors._id: 629b765ce1af194c641fcbc6
paper_authors.avatarUrl: /avatars/7c53a4c2a1e528c19641a2b601731754.svg
paper_authors.fullname: Zhipeng Chen
paper_authors.type: user
paper_authors.user: TimothyCzp
paper_authors._id: 67e5f4ad147ee85622ad0df4
paper_authors.name: Wayne Xin Zhao
paper_authors._id: 67e5f4ad147ee85622ad0df5
paper_authors.name: Zheng Liu
paper_authors._id: 67e5f4ad147ee85622ad0df6
paper_authors.name: Zhongyuan Wang
paper_authors._id: 67e5f4ad147ee85622ad0df7
paper_authors.name: Lei Fang
paper_authors._id: 67e5f4ad147ee85622ad0df8
paper_authors.name: Ji-Rong Wen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-28T08:47:20.525Z
paper_authors._id: 64b8c89052b7353d8c6a1013
paper_authors.avatarUrl: /avatars/cd59fffe81f6b07b4519540b8ff3d95f.svg
paper_authors.fullname: Ji-Rong Wen
paper_authors.type: user
paper_authors.user: jrwen
paper_title: Challenging the Boundaries of Reasoning: An Olympiad-Level Math
    Benchmark for Large Language Models
paper_summary: In recent years, the rapid development of large reasoning models has resulted
  in the saturation of existing benchmarks for evaluating mathematical reasoning,
  highlighting the urgent need for more challenging and rigorous evaluation
  frameworks. To address this gap, we introduce OlymMATH, a novel Olympiad-level
  mathematical benchmark, designed to rigorously test the complex reasoning
  capabilities of LLMs. OlymMATH features 200 meticulously curated problems, each
  manually verified and available in parallel English and Chinese versions. The
  problems are systematically organized into two distinct difficulty tiers: (1)
  AIME-level problems (easy) that establish a baseline for mathematical reasoning
  assessment, and (2) significantly more challenging problems (hard) designed to
  push the boundaries of current state-of-the-art models. In our benchmark, these
  problems span four core mathematical fields, each including a verifiable
  numerical solution to enable objective, rule-based evaluation. Empirical
  results underscore the significant challenge presented by OlymMATH, with
  state-of-the-art models including DeepSeek-R1 and OpenAI's o3-mini
  demonstrating notably limited accuracy on the hard subset. Furthermore, the
  benchmark facilitates comprehensive bilingual assessment of mathematical
  reasoning abilities-a critical dimension that remains largely unaddressed in
  mainstream mathematical reasoning benchmarks. We release the OlymMATH benchmark
  at the STILL project: https://github.com/RUCAIBox/Slow_Thinking_with_LLMs.
paper_ai_summary: OlymMATH is a new benchmark for evaluating the complex reasoning capabilities of LLMs using challenging Olympiad-level problems in parallel English and Chinese.
paper_ai_keywords: LLMs DeepSeek-R1 o3-mini bilingual assessment mathematical reasoning benchmark AIME-level problems