title: Free Process Rewards without Process Labels
summary: Different from its counterpart outcome reward models (ORMs), which evaluate
  the entire responses, a process reward model (PRM) scores a reasoning
  trajectory step by step, providing denser and more fine grained rewards.
  However, training a PRM requires labels annotated at every intermediate step,
  presenting significant challenges for both manual and automatic data
  collection. This paper aims to address this challenge. Both theoretically and
  empirically, we show that an implicit PRM can be obtained at no
  additional cost, by simply training an ORM on the cheaper response-level
  labels. The only assumption is to parameterize the outcome reward as the
  log-likelihood ratios of the policy and reference models, which can be
  optimized regardless of the specific choice of loss objectives. In experiments,
  we instantiate our implicit PRMs with various objectives and evaluate their
  performance on MATH. We show that our implicit PRM outperforms a strong
  MCTS-based baseline \'a la Math-Shepherd using less than 1/38 of the
  training data. Its performance can be further improved with majority voting. We
  further find that scaling up instructions and responses benefits our implicit
  PRM, and the latter brings a larger gain. Particularly, we find that our
  implicit PRM, when instantiated with the cross-entropy (CE) loss, is more
  data-efficient and can keep improving generation models even when trained with
  only one response per instruction, the setup that suffers from extreme data
  scarcity and imbalance. Further, instructions should be relevant to downstream
  tasks while the diversity of responses does not bring gains. Surprisingly,
  training on extra Math-Shepherd step labels brings no further improvements to
  our implicit PRM trained on only outcome data. We hope that our work will
  encourage a rethinking of PRM training approaches and contribute to making
  training PRMs more accessible.
paper_authors._id: 674fd56a16234f46d2559385
paper_authors.name: Lifan Yuan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-04T09:44:25.994Z
paper_authors._id: 61506ce0b0e47650b93f79d0
paper_authors.avatarUrl: /avatars/21301cd9929a1a64f3f6427413855ea8.svg
paper_authors.fullname: Lievan
paper_authors.type: user
paper_authors.user: lievan
paper_authors._id: 674fd56a16234f46d2559386
paper_authors.name: Wendi Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-04T11:56:09.030Z
paper_authors._id: 671bfaa29e5e675c7f5c4307
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/PwDA6OSSAmg6k4LliEQkZ.png
paper_authors.fullname: Wendi Li
paper_authors.type: user
paper_authors.user: wendili
paper_authors._id: 674fd56a16234f46d2559387
paper_authors.name: Huayu Chen
paper_authors._id: 674fd56a16234f46d2559388
paper_authors.name: Ganqu Cui
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-04T09:44:24.028Z
paper_authors._id: 650eba9555dc1e841746f132
paper_authors.avatarUrl: /avatars/af6f5ee78f161d25ec0afc45d2def8eb.svg
paper_authors.fullname: Ganqu Cui
paper_authors.type: user
paper_authors.user: ganqu
paper_authors._id: 674fd56a16234f46d2559389
paper_authors.name: Ning Ding
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-04T11:57:00.322Z
paper_authors._id: 60cf4bcb1ce3775ebb86e5d5
paper_authors.avatarUrl: /avatars/12bcd18d215abf91f297f93007733148.svg
paper_authors.fullname: Ning Ding
paper_authors.type: user
paper_authors.user: stingning
paper_authors._id: 674fd56a16234f46d255938a
paper_authors.name: Kaiyan Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-04T11:57:22.757Z
paper_authors._id: 60bc94cd85a3ab33829b6211
paper_authors.avatarUrl: /avatars/b57d36c7577fbbb42ea5b963eef4144a.svg
paper_authors.fullname: Kaiyan Zhang
paper_authors.type: user
paper_authors.user: iseesaw
paper_authors._id: 674fd56a16234f46d255938b
paper_authors.name: Bowen Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-04T11:57:28.976Z
paper_authors._id: 669f614b59adf5b56e05bce3
paper_authors.avatarUrl: /avatars/ffd4189efbceb0e63a03db273065a44b.svg
paper_authors.fullname: BowenZhou
paper_authors.type: user
paper_authors.user: bowenZhou
paper_authors._id: 674fd56a16234f46d255938c
paper_authors.name: Zhiyuan Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-04T11:57:36.966Z
paper_authors._id: 6472f1f7485a7c8e1bcf507b
paper_authors.avatarUrl: /avatars/79dca137583875191ea883b193b630b7.svg
paper_authors.fullname: Zhiyuan Liu
paper_authors.type: user
paper_authors.user: zibuyu9
paper_authors._id: 674fd56a16234f46d255938d
paper_authors.name: Hao Peng
paper_title: Free Process Rewards without Process Labels
paper_summary: Different from its counterpart outcome reward models (ORMs), which evaluate
  the entire responses, a process reward model (PRM) scores a reasoning
  trajectory step by step, providing denser and more fine grained rewards.
  However, training a PRM requires labels annotated at every intermediate step,
  presenting significant challenges for both manual and automatic data
  collection. This paper aims to address this challenge. Both theoretically and
  empirically, we show that an implicit PRM can be obtained at no
  additional cost, by simply training an ORM on the cheaper response-level
  labels. The only assumption is to parameterize the outcome reward as the
  log-likelihood ratios of the policy and reference models, which can be
  optimized regardless of the specific choice of loss objectives. In experiments,
  we instantiate our implicit PRMs with various objectives and evaluate their
  performance on MATH. We show that our implicit PRM outperforms a strong
  MCTS-based baseline \'a la Math-Shepherd using less than 1/38 of the
  training data. Its performance can be further improved with majority voting. We
  further find that scaling up instructions and responses benefits our implicit
  PRM, and the latter brings a larger gain. Particularly, we find that our
  implicit PRM, when instantiated with the cross-entropy (CE) loss, is more
  data-efficient and can keep improving generation models even when trained with
  only one response per instruction, the setup that suffers from extreme data
  scarcity and imbalance. Further, instructions should be relevant to downstream
  tasks while the diversity of responses does not bring gains. Surprisingly,
  training on extra Math-Shepherd step labels brings no further improvements to
  our implicit PRM trained on only outcome data. We hope that our work will
  encourage a rethinking of PRM training approaches and contribute to making
  training PRMs more accessible.
paper_ai_summary: An implicit process reward model can be trained without additional step labels by parameterizing outcome rewards as log-likelihood ratios, outperforming traditional methods with less data and better generalization.
paper_ai_keywords: process reward model PRM outcome reward model ORM reasoning trajectory log-likelihood ratios MCTS Math-Shepherd majority voting cross-entropy loss data efficiency extreme data scarcity