Is That Your Final Answer? Test-Time Scaling Improves Selective Question
  Answering
Scaling the test-time compute of large language models has demonstrated
impressive performance on reasoning benchmarks. However, existing evaluations
of test-time scaling make the strong assumption that a reasoning system should
always give an answer to any question provided. This overlooks concerns about
whether a model is confident in its answer, and whether it is appropriate to
always provide a response. To address these concerns, we extract confidence
scores during reasoning for thresholding model responses. We find that
increasing compute budget at inference time not only helps models answer more
questions correctly, but also increases confidence in correct responses. We
then extend the current paradigm of zero-risk responses during evaluation by
considering settings with non-zero levels of response risk, and suggest a
recipe for reporting evaluations under these settings.
67b691751f861500916ecd5d
William Jurayj
claimed_verified
2025-02-20T09:36:09.674Z
6372bc95c4267fd7cd77f4d0
/avatars/17a24af68f45487e601687d777b352b6.svg
William Jurayj
user
wjurayj
67b691751f861500916ecd5e
Jeffrey Cheng
claimed_verified
2025-02-20T15:53:07.241Z
65f28eebf5cf26fe0632ce67
/avatars/cd970cfe4215374c82d47df57ac30795.svg
Jeffrey Cheng
user
nexync
67b691751f861500916ecd5f
Benjamin Van Durme
Is That Your Final Answer? Test-Time Scaling Improves Selective Question
  Answering
Scaling the test-time compute of large language models has demonstrated
impressive performance on reasoning benchmarks. However, existing evaluations
of test-time scaling make the strong assumption that a reasoning system should
always give an answer to any question provided. This overlooks concerns about
whether a model is confident in its answer, and whether it is appropriate to
always provide a response. To address these concerns, we extract confidence
scores during reasoning for thresholding model responses. We find that
increasing compute budget at inference time not only helps models answer more
questions correctly, but also increases confidence in correct responses. We
then extend the current paradigm of zero-risk responses during evaluation by
considering settings with non-zero levels of response risk, and suggest a
recipe for reporting evaluations under these settings.
Evaluating large language models includes confidence thresholding to manage response risk at test time, showing that increased compute improves both accuracy and confidence.
confidence scores inference time zero-risk responses response risk