title: Scaling LLM Test-Time Compute Optimally can be More Effective than
    Scaling Model Parameters
summary: Enabling LLMs to improve their outputs by using more test-time computation is
  a critical step towards building generally self-improving agents that can
  operate on open-ended natural language. In this paper, we study the scaling of
  inference-time computation in LLMs, with a focus on answering the question: if
  an LLM is allowed to use a fixed but non-trivial amount of inference-time
  compute, how much can it improve its performance on a challenging prompt?
  Answering this question has implications not only on the achievable performance
  of LLMs, but also on the future of LLM pretraining and how one should tradeoff
  inference-time and pre-training compute. Despite its importance, little
  research attempted to understand the scaling behaviors of various test-time
  inference methods. Moreover, current work largely provides negative results for
  a number of these strategies. In this work, we analyze two primary mechanisms
  to scale test-time computation: (1) searching against dense, process-based
  verifier reward models; and (2) updating the model's distribution over a
  response adaptively, given the prompt at test time. We find that in both cases,
  the effectiveness of different approaches to scaling test-time compute
  critically varies depending on the difficulty of the prompt. This observation
  motivates applying a "compute-optimal" scaling strategy, which acts to most
  effectively allocate test-time compute adaptively per prompt. Using this
  compute-optimal strategy, we can improve the efficiency of test-time compute
  scaling by more than 4x compared to a best-of-N baseline. Additionally, in a
  FLOPs-matched evaluation, we find that on problems where a smaller base model
  attains somewhat non-trivial success rates, test-time compute can be used to
  outperform a 14x larger model.
paper_authors._id: 66b2d3a09f3dfa895cd036be
paper_authors.name: Charlie Snell
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-08-07T01:56:25.477Z
paper_authors._id: 605d07fa8a3450814bada877
paper_authors.avatarUrl: /avatars/eafe986982057fbaba962b99d5543477.svg
paper_authors.fullname: Charlie Snell
paper_authors.type: user
paper_authors.user: sea-snell
paper_authors._id: 66b2d3a09f3dfa895cd036bf
paper_authors.name: Jaehoon Lee
paper_authors._id: 66b2d3a09f3dfa895cd036c0
paper_authors.name: Kelvin Xu
paper_authors._id: 66b2d3a09f3dfa895cd036c1
paper_authors.name: Aviral Kumar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-08T12:21:23.161Z
paper_authors._id: 64879130cc0e54f272b5dbcc
paper_authors.avatarUrl: /avatars/ead1211a56a4e5c14cb8f6b2ea59c0e6.svg
paper_authors.fullname: Aviral Kumar
paper_authors.type: user
paper_authors.user: aviralkumar
paper_title: Scaling LLM Test-Time Compute Optimally can be More Effective than
    Scaling Model Parameters
paper_summary: Enabling LLMs to improve their outputs by using more test-time computation is
  a critical step towards building generally self-improving agents that can
  operate on open-ended natural language. In this paper, we study the scaling of
  inference-time computation in LLMs, with a focus on answering the question: if
  an LLM is allowed to use a fixed but non-trivial amount of inference-time
  compute, how much can it improve its performance on a challenging prompt?
  Answering this question has implications not only on the achievable performance
  of LLMs, but also on the future of LLM pretraining and how one should tradeoff
  inference-time and pre-training compute. Despite its importance, little
  research attempted to understand the scaling behaviors of various test-time
  inference methods. Moreover, current work largely provides negative results for
  a number of these strategies. In this work, we analyze two primary mechanisms
  to scale test-time computation: (1) searching against dense, process-based
  verifier reward models; and (2) updating the model's distribution over a
  response adaptively, given the prompt at test time. We find that in both cases,
  the effectiveness of different approaches to scaling test-time compute
  critically varies depending on the difficulty of the prompt. This observation
  motivates applying a "compute-optimal" scaling strategy, which acts to most
  effectively allocate test-time compute adaptively per prompt. Using this
  compute-optimal strategy, we can improve the efficiency of test-time compute
  scaling by more than 4x compared to a best-of-N baseline. Additionally, in a
  FLOPs-matched evaluation, we find that on problems where a smaller base model
  attains somewhat non-trivial success rates, test-time compute can be used to
  outperform a 14x larger model.
paper_ai_keywords: LLMs (Large Language Models) Inference-time computation Scalability of inference Test-time computation Prompt answering Performance improvement LLM pretraining Tradeoff between inference-time and pre-training compute Test-time inference methods Dense, process-based verifier rewards models Adaptive update of model's distribution over a response Compute-optimal scaling strategy FLOPs (Floating Point Operations Per Second)