A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths
  to Reproducibility
Reasoning has emerged as the next major frontier for language models (LMs),
with rapid advances from both academic and industrial labs. However, this
progress often outpaces methodological rigor, with many evaluations relying on
benchmarking practices that lack transparency, robustness, or statistical
grounding. In this work, we conduct a comprehensive empirical study and find
that current mathematical reasoning benchmarks are highly sensitive to subtle
implementation choices - including decoding parameters, random seeds, prompt
formatting, and even hardware and software-framework configurations.
Performance gains reported in recent studies frequently hinge on unclear
comparisons or unreported sources of variance. To address these issues, we
propose a standardized evaluation framework with clearly defined best practices
and reporting standards. Using this framework, we reassess recent methods and
find that reinforcement learning (RL) approaches yield only modest improvements
- far below prior claims - and are prone to overfitting, especially on
small-scale benchmarks like AIME24. In contrast, supervised finetuning (SFT)
methods show consistently stronger generalization. To foster reproducibility,
we release all code, prompts, and model outputs, for reasoning benchmarks,
establishing more rigorous foundations for future work.
67f75609b2d783993db63aba
Andreas Hochlehnert
admin_assigned
2025-04-10T08:07:58.732Z
64ff3944f0d65cca9b867ed2
https://cdn-avatars.huggingface.co/v1/production/uploads/64ff3944f0d65cca9b867ed2/jWnHkF4AUzh51MkC0UT6b.png
Andreas Hochlehnert
user
libeanim
67f75609b2d783993db63abb
Hardik Bhatnagar
admin_assigned
2025-04-10T08:08:04.922Z
6556760b35f26c82c09a010f
https://cdn-avatars.huggingface.co/v1/production/uploads/6556760b35f26c82c09a010f/hNbwvXRBsKo6pqrHbXNzz.jpeg
Hardik Bhatnagar
user
hrdkbhatnagar
67f75609b2d783993db63abc
Vishaal Udandarao
admin_assigned
2025-04-10T08:08:10.717Z
6304da46ce6b12280b1bd575
https://cdn-avatars.huggingface.co/v1/production/uploads/6304da46ce6b12280b1bd575/V96ocKW4HOoysAGxuAH1X.jpeg
Vishaal Udandarao
user
vishaal27
67f75609b2d783993db63abd
Samuel Albanie
admin_assigned
2025-04-10T08:08:16.809Z
62f3efefd6ba2ee26651f44a
https://cdn-avatars.huggingface.co/v1/production/uploads/1660153837083-noauth.png
Samuel Albanie
user
albanie
67f75609b2d783993db63abe
Ameya Prabhu
claimed_verified
2025-04-10T06:39:24.192Z
6464a0d41683d3c81f51924a
https://cdn-avatars.huggingface.co/v1/production/uploads/6464a0d41683d3c81f51924a/s7yYVwfUB4WOhVFJS6A6T.jpeg
Ameya Prabhu
user
AmeyaPrabhu
67f75609b2d783993db63abf
Matthias Bethge
A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths
  to Reproducibility
Reasoning has emerged as the next major frontier for language models (LMs),
with rapid advances from both academic and industrial labs. However, this
progress often outpaces methodological rigor, with many evaluations relying on
benchmarking practices that lack transparency, robustness, or statistical
grounding. In this work, we conduct a comprehensive empirical study and find
that current mathematical reasoning benchmarks are highly sensitive to subtle
implementation choices - including decoding parameters, random seeds, prompt
formatting, and even hardware and software-framework configurations.
Performance gains reported in recent studies frequently hinge on unclear
comparisons or unreported sources of variance. To address these issues, we
propose a standardized evaluation framework with clearly defined best practices
and reporting standards. Using this framework, we reassess recent methods and
find that reinforcement learning (RL) approaches yield only modest improvements
- far below prior claims - and are prone to overfitting, especially on
small-scale benchmarks like AIME24. In contrast, supervised finetuning (SFT)
methods show consistently stronger generalization. To foster reproducibility,
we release all code, prompts, and model outputs, for reasoning benchmarks,
establishing more rigorous foundations for future work.
A standardized evaluation framework reveals that reinforcement learning offers limited improvements over supervised fine-tuning in mathematical reasoning benchmarks, highlighting issues with transparency and variability in current methodologies.
reinforcement learning supervised finetuning mathematical reasoning benchmarks evaluation framework decoding parameters random seeds prompt formatting hardware software-framework configurations overfitting AIME24 reproducibility