title: Is That Your Final Answer? Test-Time Scaling Improves Selective Question
    Answering
summary: Scaling the test-time compute of large language models has demonstrated
  impressive performance on reasoning benchmarks. However, existing evaluations
  of test-time scaling make the strong assumption that a reasoning system should
  always give an answer to any question provided. This overlooks concerns about
  whether a model is confident in its answer, and whether it is appropriate to
  always provide a response. To address these concerns, we extract confidence
  scores during reasoning for thresholding model responses. We find that
  increasing compute budget at inference time not only helps models answer more
  questions correctly, but also increases confidence in correct responses. We
  then extend the current paradigm of zero-risk responses during evaluation by
  considering settings with non-zero levels of response risk, and suggest a
  recipe for reporting evaluations under these settings.
paper_authors._id: 67b691751f861500916ecd5d
paper_authors.name: William Jurayj
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-20T09:36:09.674Z
paper_authors._id: 6372bc95c4267fd7cd77f4d0
paper_authors.avatarUrl: /avatars/17a24af68f45487e601687d777b352b6.svg
paper_authors.fullname: William Jurayj
paper_authors.type: user
paper_authors.user: wjurayj
paper_authors._id: 67b691751f861500916ecd5e
paper_authors.name: Jeffrey Cheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-20T15:53:07.241Z
paper_authors._id: 65f28eebf5cf26fe0632ce67
paper_authors.avatarUrl: /avatars/cd970cfe4215374c82d47df57ac30795.svg
paper_authors.fullname: Jeffrey Cheng
paper_authors.type: user
paper_authors.user: nexync
paper_authors._id: 67b691751f861500916ecd5f
paper_authors.name: Benjamin Van Durme
paper_title: Is That Your Final Answer? Test-Time Scaling Improves Selective Question
    Answering
paper_summary: Scaling the test-time compute of large language models has demonstrated
  impressive performance on reasoning benchmarks. However, existing evaluations
  of test-time scaling make the strong assumption that a reasoning system should
  always give an answer to any question provided. This overlooks concerns about
  whether a model is confident in its answer, and whether it is appropriate to
  always provide a response. To address these concerns, we extract confidence
  scores during reasoning for thresholding model responses. We find that
  increasing compute budget at inference time not only helps models answer more
  questions correctly, but also increases confidence in correct responses. We
  then extend the current paradigm of zero-risk responses during evaluation by
  considering settings with non-zero levels of response risk, and suggest a
  recipe for reporting evaluations under these settings.
paper_ai_keywords: confidence scores test-time compute large language models reasoning benchmarks zer-risk responses