The Relationship Between Reasoning and Performance in Large Language
  Models -- o3 (mini) Thinks Harder, Not Longer
Large language models have demonstrated remarkable progress in mathematical
reasoning, leveraging chain-of-thought and test-time compute scaling. However,
many open questions remain regarding the interplay between reasoning token
usage and accuracy gains. In particular, when comparing models across
generations, it is unclear whether improved performance results from longer
reasoning chains or more efficient reasoning. We systematically analyze
chain-of-thought length across o1-mini and o3-mini variants on the Omni-MATH
benchmark, finding that o3-mini (m) achieves superior accuracy without
requiring longer reasoning chains than o1-mini. Moreover, we show that accuracy
generally declines as reasoning chains grow across all models and compute
settings, even when controlling for difficulty of the questions. This accuracy
drop is significantly smaller in more proficient models, suggesting that new
generations of reasoning models use test-time compute more effectively.
Finally, we highlight that while o3-mini (h) achieves a marginal accuracy gain
over o3-mini (m), it does so by allocating substantially more reasoning tokens
across all problems, even the ones that o3-mini (m) can already solve. These
findings provide new insights into the relationship between model capability
and reasoning length, with implications for efficiency, scaling, and evaluation
methodologies.
67bbdbe8ea3003f47f15d036
Marthe Ballon
extracted_pending
2025-06-25T15:11:05.334Z
685c115b18f9b3926bcf1070
/avatars/1051de2673e0aa9378d31c829043ad13.svg
Marthe Ballon
user
martheballon
67bbdbe8ea3003f47f15d037
Andres Algaba
67bbdbe8ea3003f47f15d038
Vincent Ginis
admin_assigned
2025-02-24T15:53:02.578Z
62d69d324be637062ecd88de
/avatars/445cfce9170484d32cfa379015d9cd14.svg
Vincent Ginis
user
VincentGinis
The Relationship Between Reasoning and Performance in Large Language
  Models -- o3 (mini) Thinks Harder, Not Longer
Large language models have demonstrated remarkable progress in mathematical
reasoning, leveraging chain-of-thought and test-time compute scaling. However,
many open questions remain regarding the interplay between reasoning token
usage and accuracy gains. In particular, when comparing models across
generations, it is unclear whether improved performance results from longer
reasoning chains or more efficient reasoning. We systematically analyze
chain-of-thought length across o1-mini and o3-mini variants on the Omni-MATH
benchmark, finding that o3-mini (m) achieves superior accuracy without
requiring longer reasoning chains than o1-mini. Moreover, we show that accuracy
generally declines as reasoning chains grow across all models and compute
settings, even when controlling for difficulty of the questions. This accuracy
drop is significantly smaller in more proficient models, suggesting that new
generations of reasoning models use test-time compute more effectively.
Finally, we highlight that while o3-mini (h) achieves a marginal accuracy gain
over o3-mini (m), it does so by allocating substantially more reasoning tokens
across all problems, even the ones that o3-mini (m) can already solve. These
findings provide new insights into the relationship between model capability
and reasoning length, with implications for efficiency, scaling, and evaluation
methodologies.
Analysis reveals that newer large language models achieve higher accuracy without longer reasoning chains, indicating more efficient use of computational resources.
chain-of-thought test-time compute scaling Omni-MATH benchmark reasoning token usage o1-mini o3-mini reasoning chains model capability computational resources evaluation methodologies