title: The Relationship Between Reasoning and Performance in Large Language
    Models -- o3 (mini) Thinks Harder, Not Longer
summary: Large language models have demonstrated remarkable progress in mathematical
  reasoning, leveraging chain-of-thought and test-time compute scaling. However,
  many open questions remain regarding the interplay between reasoning token
  usage and accuracy gains. In particular, when comparing models across
  generations, it is unclear whether improved performance results from longer
  reasoning chains or more efficient reasoning. We systematically analyze
  chain-of-thought length across o1-mini and o3-mini variants on the Omni-MATH
  benchmark, finding that o3-mini (m) achieves superior accuracy without
  requiring longer reasoning chains than o1-mini. Moreover, we show that accuracy
  generally declines as reasoning chains grow across all models and compute
  settings, even when controlling for difficulty of the questions. This accuracy
  drop is significantly smaller in more proficient models, suggesting that new
  generations of reasoning models use test-time compute more effectively.
  Finally, we highlight that while o3-mini (h) achieves a marginal accuracy gain
  over o3-mini (m), it does so by allocating substantially more reasoning tokens
  across all problems, even the ones that o3-mini (m) can already solve. These
  findings provide new insights into the relationship between model capability
  and reasoning length, with implications for efficiency, scaling, and evaluation
  methodologies.
paper_authors._id: 67bbdbe8ea3003f47f15d036
paper_authors.name: Marthe Ballon
paper_authors._id: 67bbdbe8ea3003f47f15d037
paper_authors.name: Andres Algaba
paper_authors._id: 67bbdbe8ea3003f47f15d038
paper_authors.name: Vincent Ginis
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-24T15:53:02.578Z
paper_authors._id: 62d69d324be637062ecd88de
paper_authors.avatarUrl: /avatars/445cfce9170484d32cfa379015d9cd14.svg
paper_authors.fullname: Vincent Ginis
paper_authors.type: user
paper_authors.user: VincentGinis
paper_title: The Relationship Between Reasoning and Performance in Large Language
    Models -- o3 (mini) Thinks Harder, Not Longer
paper_summary: Large language models have demonstrated remarkable progress in mathematical
  reasoning, leveraging chain-of-thought and test-time compute scaling. However,
  many open questions remain regarding the interplay between reasoning token
  usage and accuracy gains. In particular, when comparing models across
  generations, it is unclear whether improved performance results from longer
  reasoning chains or more efficient reasoning. We systematically analyze
  chain-of-thought length across o1-mini and o3-mini variants on the Omni-MATH
  benchmark, finding that o3-mini (m) achieves superior accuracy without
  requiring longer reasoning chains than o1-mini. Moreover, we show that accuracy
  generally declines as reasoning chains grow across all models and compute
  settings, even when controlling for difficulty of the questions. This accuracy
  drop is significantly smaller in more proficient models, suggesting that new
  generations of reasoning models use test-time compute more effectively.
  Finally, we highlight that while o3-mini (h) achieves a marginal accuracy gain
  over o3-mini (m), it does so by allocating substantially more reasoning tokens
  across all problems, even the ones that o3-mini (m) can already solve. These
  findings provide new insights into the relationship between model capability
  and reasoning length, with implications for efficiency, scaling, and evaluation
  methodologies.
paper_ai_summary: Analysis reveals that newer large language models achieve higher accuracy without longer reasoning chains, indicating more efficient use of computational resources.
paper_ai_keywords: chain-of-thought test-time compute scaling Omni-MATH benchmark reasoning token usage o1-mini o3-mini reasoning chains model capability computational resources evaluation methodologies