title: THOUGHTTERMINATOR: Benchmarking, Calibrating, and Mitigating
    Overthinking in Reasoning Models
summary: Reasoning models have demonstrated impressive performance on difficult tasks
  that traditional language models struggle at. However, many are plagued with
  the problem of overthinking--generating large amounts of unnecessary tokens
  which don't improve accuracy on a question. We introduce approximate measures
  of problem-level difficulty and demonstrate that a clear relationship between
  problem difficulty and optimal token spend exists, and evaluate how well
  calibrated a variety of reasoning models are in terms of efficiently allocating
  the optimal token count. We find that in general, reasoning models are poorly
  calibrated, particularly on easy problems. To evaluate calibration on easy
  questions we introduce DUMB500, a dataset of extremely easy math, reasoning,
  code, and task problems, and jointly evaluate reasoning model on these simple
  examples and extremely difficult examples from existing frontier benchmarks on
  the same task domain. Finally, we introduce THOUGHTTERMINATOR, a training-free
  black box decoding technique that significantly improves reasoning model
  calibration.
paper_authors._id: 68072cae860bab681fa8c2c9
paper_authors.name: Xiao Pu
paper_authors._id: 68072cae860bab681fa8c2ca
paper_authors.name: Michael Saxon
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-04-22T05:44:17.040Z
paper_authors._id: 6002c1db698168af3bb9f4a5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1610793426015-noauth.png
paper_authors.fullname: Michael Saxon
paper_authors.type: user
paper_authors.user: saxon
paper_authors._id: 68072cae860bab681fa8c2cb
paper_authors.name: Wenyue Hua
paper_authors._id: 68072cae860bab681fa8c2cc
paper_authors.name: William Yang Wang
paper_title: THOUGHTTERMINATOR: Benchmarking, Calibrating, and Mitigating
    Overthinking in Reasoning Models
paper_summary: Reasoning models have demonstrated impressive performance on difficult tasks
  that traditional language models struggle at. However, many are plagued with
  the problem of overthinking--generating large amounts of unnecessary tokens
  which don't improve accuracy on a question. We introduce approximate measures
  of problem-level difficulty and demonstrate that a clear relationship between
  problem difficulty and optimal token spend exists, and evaluate how well
  calibrated a variety of reasoning models are in terms of efficiently allocating
  the optimal token count. We find that in general, reasoning models are poorly
  calibrated, particularly on easy problems. To evaluate calibration on easy
  questions we introduce DUMB500, a dataset of extremely easy math, reasoning,
  code, and task problems, and jointly evaluate reasoning model on these simple
  examples and extremely difficult examples from existing frontier benchmarks on
  the same task domain. Finally, we introduce THOUGHTTERMINATOR, a training-free
  black box decoding technique that significantly improves reasoning model
  calibration.
paper_ai_summary: THOUGHTTERMINATOR, a training-free decoding technique, improves the calibration of reasoning models by optimizing token usage across easy and difficult problems.
paper_ai_keywords: reasoning models token spend problem difficulty calibration DUMB500 THOUGHTTERMINATOR