THOUGHTTERMINATOR: Benchmarking, Calibrating, and Mitigating
  Overthinking in Reasoning Models
Reasoning models have demonstrated impressive performance on difficult tasks
that traditional language models struggle at. However, many are plagued with
the problem of overthinking--generating large amounts of unnecessary tokens
which don't improve accuracy on a question. We introduce approximate measures
of problem-level difficulty and demonstrate that a clear relationship between
problem difficulty and optimal token spend exists, and evaluate how well
calibrated a variety of reasoning models are in terms of efficiently allocating
the optimal token count. We find that in general, reasoning models are poorly
calibrated, particularly on easy problems. To evaluate calibration on easy
questions we introduce DUMB500, a dataset of extremely easy math, reasoning,
code, and task problems, and jointly evaluate reasoning model on these simple
examples and extremely difficult examples from existing frontier benchmarks on
the same task domain. Finally, we introduce THOUGHTTERMINATOR, a training-free
black box decoding technique that significantly improves reasoning model
calibration.
68072cae860bab681fa8c2c9
Xiao Pu
68072cae860bab681fa8c2ca
Michael Saxon
extracted_pending
2025-04-22T05:44:17.040Z
6002c1db698168af3bb9f4a5
https://cdn-avatars.huggingface.co/v1/production/uploads/6002c1db698168af3bb9f4a5/IO4sXemgIhKzib7CfeWAC.jpeg
Michael Saxon
user
saxon
68072cae860bab681fa8c2cb
Wenyue Hua
68072cae860bab681fa8c2cc
William Yang Wang
THOUGHTTERMINATOR: Benchmarking, Calibrating, and Mitigating
  Overthinking in Reasoning Models
Reasoning models have demonstrated impressive performance on difficult tasks
that traditional language models struggle at. However, many are plagued with
the problem of overthinking--generating large amounts of unnecessary tokens
which don't improve accuracy on a question. We introduce approximate measures
of problem-level difficulty and demonstrate that a clear relationship between
problem difficulty and optimal token spend exists, and evaluate how well
calibrated a variety of reasoning models are in terms of efficiently allocating
the optimal token count. We find that in general, reasoning models are poorly
calibrated, particularly on easy problems. To evaluate calibration on easy
questions we introduce DUMB500, a dataset of extremely easy math, reasoning,
code, and task problems, and jointly evaluate reasoning model on these simple
examples and extremely difficult examples from existing frontier benchmarks on
the same task domain. Finally, we introduce THOUGHTTERMINATOR, a training-free
black box decoding technique that significantly improves reasoning model
calibration.
THOUGHTTERMINATOR, a training-free decoding technique, improves the calibration of reasoning models by optimizing token usage across easy and difficult problems.
reasoning models token spend problem difficulty calibration DUMB500 THOUGHTTERMINATOR