title: A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths
    to Reproducibility
summary: Reasoning has emerged as the next major frontier for language models (LMs),
  with rapid advances from both academic and industrial labs. However, this
  progress often outpaces methodological rigor, with many evaluations relying on
  benchmarking practices that lack transparency, robustness, or statistical
  grounding. In this work, we conduct a comprehensive empirical study and find
  that current mathematical reasoning benchmarks are highly sensitive to subtle
  implementation choices - including decoding parameters, random seeds, prompt
  formatting, and even hardware and software-framework configurations.
  Performance gains reported in recent studies frequently hinge on unclear
  comparisons or unreported sources of variance. To address these issues, we
  propose a standardized evaluation framework with clearly defined best practices
  and reporting standards. Using this framework, we reassess recent methods and
  find that reinforcement learning (RL) approaches yield only modest improvements
  - far below prior claims - and are prone to overfitting, especially on
  small-scale benchmarks like AIME24. In contrast, supervised finetuning (SFT)
  methods show consistently stronger generalization. To foster reproducibility,
  we release all code, prompts, and model outputs, for reasoning benchmarks,
  establishing more rigorous foundations for future work.
paper_authors._id: 67f75609b2d783993db63aba
paper_authors.name: Andreas Hochlehnert
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-10T08:07:58.732Z
paper_authors._id: 64ff3944f0d65cca9b867ed2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64ff3944f0d65cca9b867ed2/jWnHkF4AUzh51MkC0UT6b.png
paper_authors.fullname: Andreas Hochlehnert
paper_authors.type: user
paper_authors.user: libeanim
paper_authors._id: 67f75609b2d783993db63abb
paper_authors.name: Hardik Bhatnagar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-10T08:08:04.922Z
paper_authors._id: 6556760b35f26c82c09a010f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6556760b35f26c82c09a010f/hNbwvXRBsKo6pqrHbXNzz.jpeg
paper_authors.fullname: Hardik Bhatnagar
paper_authors.type: user
paper_authors.user: hrdkbhatnagar
paper_authors._id: 67f75609b2d783993db63abc
paper_authors.name: Vishaal Udandarao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-10T08:08:10.717Z
paper_authors._id: 6304da46ce6b12280b1bd575
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6304da46ce6b12280b1bd575/V96ocKW4HOoysAGxuAH1X.jpeg
paper_authors.fullname: Vishaal Udandarao
paper_authors.type: user
paper_authors.user: vishaal27
paper_authors._id: 67f75609b2d783993db63abd
paper_authors.name: Samuel Albanie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-10T08:08:16.809Z
paper_authors._id: 62f3efefd6ba2ee26651f44a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1660153837083-noauth.png
paper_authors.fullname: Samuel Albanie
paper_authors.type: user
paper_authors.user: albanie
paper_authors._id: 67f75609b2d783993db63abe
paper_authors.name: Ameya Prabhu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-10T06:39:24.192Z
paper_authors._id: 6464a0d41683d3c81f51924a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6464a0d41683d3c81f51924a/s7yYVwfUB4WOhVFJS6A6T.jpeg
paper_authors.fullname: Ameya Prabhu
paper_authors.type: user
paper_authors.user: AmeyaPrabhu
paper_authors._id: 67f75609b2d783993db63abf
paper_authors.name: Matthias Bethge
paper_title: A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths
    to Reproducibility
paper_summary: Reasoning has emerged as the next major frontier for language models (LMs),
  with rapid advances from both academic and industrial labs. However, this
  progress often outpaces methodological rigor, with many evaluations relying on
  benchmarking practices that lack transparency, robustness, or statistical
  grounding. In this work, we conduct a comprehensive empirical study and find
  that current mathematical reasoning benchmarks are highly sensitive to subtle
  implementation choices - including decoding parameters, random seeds, prompt
  formatting, and even hardware and software-framework configurations.
  Performance gains reported in recent studies frequently hinge on unclear
  comparisons or unreported sources of variance. To address these issues, we
  propose a standardized evaluation framework with clearly defined best practices
  and reporting standards. Using this framework, we reassess recent methods and
  find that reinforcement learning (RL) approaches yield only modest improvements
  - far below prior claims - and are prone to overfitting, especially on
  small-scale benchmarks like AIME24. In contrast, supervised finetuning (SFT)
  methods show consistently stronger generalization. To foster reproducibility,
  we release all code, prompts, and model outputs, for reasoning benchmarks,
  establishing more rigorous foundations for future work.
paper_ai_summary: A standardized evaluation framework reveals that reinforcement learning offers limited improvements over supervised fine-tuning in mathematical reasoning benchmarks, highlighting issues with transparency and variability in current methodologies.
paper_ai_keywords: reinforcement learning supervised finetuning mathematical reasoning benchmarks evaluation framework decoding parameters random seeds prompt formatting hardware software-framework configurations overfitting AIME24 reproducibility