title: Putting the Value Back in RL: Better Test-Time Scaling by Unifying LLM
    Reasoners With Verifiers
summary: Prevalent reinforcement learning~(RL) methods for fine-tuning LLM reasoners,
  such as GRPO or Leave-one-out PPO, abandon the learned value function in favor
  of empirically estimated returns. This hinders test-time compute scaling that
  relies on using the value-function for verification. In this work, we propose
  RL^V that augments any ``value-free'' RL method by jointly training the LLM
  as both a reasoner and a generative verifier using RL-generated data, adding
  verification capabilities without significant overhead. Empirically, RL^V
  boosts MATH accuracy by over 20\% with parallel sampling and enables
  8-32times efficient test-time compute scaling compared to the base RL
  method. RL^V also exhibits strong generalization capabilities for both
  easy-to-hard and out-of-domain tasks. Furthermore, RL^V achieves
  1.2-1.6times higher performance when jointly scaling parallel and sequential
  test-time compute with a long reasoning R1 model.
paper_authors._id: 681e4de2f2cba17ad23d9f08
paper_authors.name: Kusha Sareen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-12T06:50:51.576Z
paper_authors._id: 6684206a55fa08351af543f6
paper_authors.avatarUrl: /avatars/9f706cf4c1f6ea4899a4d951758834ad.svg
paper_authors.fullname: Kusha Sareen
paper_authors.type: user
paper_authors.user: kushasareen
paper_authors._id: 681e4de2f2cba17ad23d9f09
paper_authors.name: Morgane M Moss
paper_authors._id: 681e4de2f2cba17ad23d9f0a
paper_authors.name: Alessandro Sordoni
paper_authors._id: 681e4de2f2cba17ad23d9f0b
paper_authors.name: Rishabh Agarwal
paper_authors._id: 681e4de2f2cba17ad23d9f0c
paper_authors.name: Arian Hosseini
paper_title: Putting the Value Back in RL: Better Test-Time Scaling by Unifying LLM
    Reasoners With Verifiers
paper_summary: Prevalent reinforcement learning~(RL) methods for fine-tuning LLM reasoners,
  such as GRPO or Leave-one-out PPO, abandon the learned value function in favor
  of empirically estimated returns. This hinders test-time compute scaling that
  relies on using the value-function for verification. In this work, we propose
  RL^V that augments any ``value-free'' RL method by jointly training the LLM
  as both a reasoner and a generative verifier using RL-generated data, adding
  verification capabilities without significant overhead. Empirically, RL^V
  boosts MATH accuracy by over 20\% with parallel sampling and enables
  8-32times efficient test-time compute scaling compared to the base RL
  method. RL^V also exhibits strong generalization capabilities for both
  easy-to-hard and out-of-domain tasks. Furthermore, RL^V achieves
  1.2-1.6times higher performance when jointly scaling parallel and sequential
  test-time compute with a long reasoning R1 model.
paper_ai_summary: RL$^V$ enhances LLM reasoners by integrating verification capabilities, improving MATH accuracy and enabling efficient test-time compute scaling.
paper_ai_keywords: reinforcement learning RL GRPO Leave-one-out PPO value function generative verifier RL-generated data MATH accuracy parallel sampling test-time compute scaling generalization capabilities long reasoning model