title: Optimizing Test-Time Compute via Meta Reinforcement Fine-Tuning
summary: Training models to effectively use test-time compute is crucial for improving
  the reasoning performance of LLMs. Current methods mostly do so via fine-tuning
  on search traces or running RL with 0/1 outcome reward, but do these approaches
  efficiently utilize test-time compute? Would these approaches continue to scale
  as the budget improves? In this paper, we try to answer these questions. We
  formalize the problem of optimizing test-time compute as a meta-reinforcement
  learning (RL) problem, which provides a principled perspective on spending
  test-time compute. This perspective enables us to view the long output stream
  from the LLM as consisting of several episodes run at test time and leads us to
  use a notion of cumulative regret over output tokens as a way to measure the
  efficacy of test-time compute. Akin to how RL algorithms can best tradeoff
  exploration and exploitation over training, minimizing cumulative regret would
  also provide the best balance between exploration and exploitation in the token
  stream. While we show that state-of-the-art models do not minimize regret, one
  can do so by maximizing a dense reward bonus in conjunction with the outcome
  0/1 reward RL. This bonus is the ''progress'' made by each subsequent block in
  the output stream, quantified by the change in the likelihood of eventual
  success. Using these insights, we develop Meta Reinforcement Fine-Tuning, or
  MRT, a new class of fine-tuning methods for optimizing test-time compute. MRT
  leads to a 2-3x relative gain in performance and roughly a 1.5x gain in token
  efficiency for math reasoning compared to outcome-reward RL.
paper_authors._id: 67d0e38171b6b577dbb8c72c
paper_authors.name: Yuxiao Qu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-12T08:41:33.926Z
paper_authors._id: 6500bbf5e102da55f9ed43fc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6500bbf5e102da55f9ed43fc/QZ6EAFV2CStFsILmTJw5D.jpeg
paper_authors.fullname: Yuxiao Qu
paper_authors.type: user
paper_authors.user: CohenQu
paper_authors._id: 67d0e38171b6b577dbb8c72d
paper_authors.name: Matthew Y. R. Yang
paper_authors._id: 67d0e38171b6b577dbb8c72e
paper_authors.name: Amrith Setlur
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-12T15:54:33.962Z
paper_authors._id: 652b3cf8802e3d1a4fb4669d
paper_authors.avatarUrl: /avatars/706f51c14193c2ab1bc66f48af8a62da.svg
paper_authors.fullname: Amrith Setlur
paper_authors.type: user
paper_authors.user: ars22
paper_authors._id: 67d0e38171b6b577dbb8c72f
paper_authors.name: Lewis Tunstall
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-12T15:53:47.844Z
paper_authors._id: 5f0c746619cb630495b814fd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1594651707950-noauth.jpeg
paper_authors.fullname: Lewis Tunstall
paper_authors.type: user
paper_authors.user: lewtun
paper_authors._id: 67d0e38171b6b577dbb8c730
paper_authors.name: Edward Emanuel Beeching
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-12T15:54:21.375Z
paper_authors._id: 6200d0a443eb0913fa2df7cc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1644220542819-noauth.jpeg
paper_authors.fullname: Edward Beeching
paper_authors.type: user
paper_authors.user: edbeeching
paper_authors._id: 67d0e38171b6b577dbb8c731
paper_authors.name: Ruslan Salakhutdinov
paper_authors._id: 67d0e38171b6b577dbb8c732
paper_authors.name: Aviral Kumar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-12T15:54:05.568Z
paper_authors._id: 64879130cc0e54f272b5dbcc
paper_authors.avatarUrl: /avatars/ead1211a56a4e5c14cb8f6b2ea59c0e6.svg
paper_authors.fullname: Aviral Kumar
paper_authors.type: user
paper_authors.user: aviralkumar
paper_title: Optimizing Test-Time Compute via Meta Reinforcement Fine-Tuning
paper_summary: Training models to effectively use test-time compute is crucial for improving
  the reasoning performance of LLMs. Current methods mostly do so via fine-tuning
  on search traces or running RL with 0/1 outcome reward, but do these approaches
  efficiently utilize test-time compute? Would these approaches continue to scale
  as the budget improves? In this paper, we try to answer these questions. We
  formalize the problem of optimizing test-time compute as a meta-reinforcement
  learning (RL) problem, which provides a principled perspective on spending
  test-time compute. This perspective enables us to view the long output stream
  from the LLM as consisting of several episodes run at test time and leads us to
  use a notion of cumulative regret over output tokens as a way to measure the
  efficacy of test-time compute. Akin to how RL algorithms can best tradeoff
  exploration and exploitation over training, minimizing cumulative regret would
  also provide the best balance between exploration and exploitation in the token
  stream. While we show that state-of-the-art models do not minimize regret, one
  can do so by maximizing a dense reward bonus in conjunction with the outcome
  0/1 reward RL. This bonus is the ''progress'' made by each subsequent block in
  the output stream, quantified by the change in the likelihood of eventual
  success. Using these insights, we develop Meta Reinforcement Fine-Tuning, or
  MRT, a new class of fine-tuning methods for optimizing test-time compute. MRT
  leads to a 2-3x relative gain in performance and roughly a 1.5x gain in token
  efficiency for math reasoning compared to outcome-reward RL.
paper_ai_keywords: meta-reinforcement learning (RL) cumulative regret token stream exploration and exploitation dense reward bonus likelihood of eventual success Meta Reinforcement Fine-Tuning (MRT)