title: Process Reinforcement through Implicit Rewards
summary: Dense process rewards have proven a more effective alternative to the sparse
  outcome-level rewards in the inference-time scaling of large language models
  (LLMs), particularly in tasks requiring complex multi-step reasoning. While
  dense rewards also offer an appealing choice for the reinforcement learning
  (RL) of LLMs since their fine-grained rewards have the potential to address
  some inherent issues of outcome rewards, such as training efficiency and credit
  assignment, this potential remains largely unrealized. This can be primarily
  attributed to the challenges of training process reward models (PRMs) online,
  where collecting high-quality process labels is prohibitively expensive, making
  them particularly vulnerable to reward hacking. To address these challenges, we
  propose PRIME (Process Reinforcement through IMplicit rEwards), which enables
  online PRM updates using only policy rollouts and outcome labels through
  implict process rewards. PRIME combines well with various advantage functions
  and forgoes the dedicated reward model training phrase that existing approaches
  require, substantially reducing the development overhead. We demonstrate
  PRIME's effectiveness on competitional math and coding. Starting from
  Qwen2.5-Math-7B-Base, PRIME achieves a 15.1% average improvement across several
  key reasoning benchmarks over the SFT model. Notably, our resulting model,
  Eurus-2-7B-PRIME, surpasses Qwen2.5-Math-7B-Instruct on seven reasoning
  benchmarks with 10% of its training data.
paper_authors._id: 67a19d705efa4fab15497775
paper_authors.name: Ganqu Cui
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-04T09:39:23.889Z
paper_authors._id: 650eba9555dc1e841746f132
paper_authors.avatarUrl: /avatars/af6f5ee78f161d25ec0afc45d2def8eb.svg
paper_authors.fullname: Ganqu Cui
paper_authors.type: user
paper_authors.user: ganqu
paper_authors._id: 67a19d705efa4fab15497776
paper_authors.name: Lifan Yuan
paper_authors._id: 67a19d705efa4fab15497777
paper_authors.name: Zefan Wang
paper_authors._id: 67a19d705efa4fab15497778
paper_authors.name: Hanbin Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-04T09:39:25.869Z
paper_authors._id: 6321152b8c0da827c72c7c16
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1678783813705-6321152b8c0da827c72c7c16.jpeg
paper_authors.fullname: Hanbin Wang
paper_authors.type: user
paper_authors.user: hanbin
paper_authors._id: 67a19d705efa4fab15497779
paper_authors.name: Wendi Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-04T14:51:24.261Z
paper_authors._id: 671bfaa29e5e675c7f5c4307
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/PwDA6OSSAmg6k4LliEQkZ.png
paper_authors.fullname: Wendi Li
paper_authors.type: user
paper_authors.user: wendili
paper_authors._id: 67a19d705efa4fab1549777a
paper_authors.name: Bingxiang He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-04T14:51:31.090Z
paper_authors._id: 64c5e944979493279b700cb2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/vjFuPWw8Vl7b7gXB19Sk-.jpeg
paper_authors.fullname: Bingxiang He
paper_authors.type: user
paper_authors.user: hbx
paper_authors._id: 67a19d705efa4fab1549777b
paper_authors.name: Yuchen Fan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-04T14:51:56.597Z
paper_authors._id: 672c2d7816766a76a747b7b5
paper_authors.avatarUrl: /avatars/12c7b26d2b81721ccac3a5c71e32a1a1.svg
paper_authors.fullname: Yuchen Fan
paper_authors.type: user
paper_authors.user: yuchenFan
paper_authors._id: 67a19d705efa4fab1549777c
paper_authors.name: Tianyu Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-04T14:52:26.615Z
paper_authors._id: 64abc4aa6cadc7aca585dddf
paper_authors.avatarUrl: /avatars/736afea979cd0021c7a37f68731524ea.svg
paper_authors.fullname: Tianyu Yu
paper_authors.type: user
paper_authors.user: Yirany
paper_authors._id: 67a19d705efa4fab1549777d
paper_authors.name: Qixin Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-06T14:15:22.453Z
paper_authors._id: 6680f0b20b72be136708af26
paper_authors.avatarUrl: /avatars/5d8fd5be0cf94e246b46abb9d3cc8f5c.svg
paper_authors.fullname: XuQixin
paper_authors.type: user
paper_authors.user: Racktic
paper_authors._id: 67a19d705efa4fab1549777e
paper_authors.name: Weize Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-04T14:52:46.343Z
paper_authors._id: 648312243b7fe59c876c0dca
paper_authors.avatarUrl: /avatars/c26ad76cd213529e4670bb599b8199bb.svg
paper_authors.fullname: weize
paper_authors.type: user
paper_authors.user: weizechen
paper_authors._id: 67a19d705efa4fab1549777f
paper_authors.name: Jiarui Yuan
paper_authors._id: 67a19d705efa4fab15497780
paper_authors.name: Huayu Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-04T14:53:06.620Z
paper_authors._id: 6630f87ee53fcb71c3887df0
paper_authors.avatarUrl: /avatars/50191a3d45bebf90cf08df09477e95db.svg
paper_authors.fullname: HuayuChen
paper_authors.type: user
paper_authors.user: HuayuChen
paper_authors._id: 67a19d705efa4fab15497781
paper_authors.name: Kaiyan Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-11T15:36:03.829Z
paper_authors._id: 60bc94cd85a3ab33829b6211
paper_authors.avatarUrl: /avatars/b57d36c7577fbbb42ea5b963eef4144a.svg
paper_authors.fullname: Kaiyan Zhang
paper_authors.type: user
paper_authors.user: iseesaw
paper_authors._id: 67a19d705efa4fab15497782
paper_authors.name: Xingtai Lv
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-04T14:53:21.172Z
paper_authors._id: 663f07d029be04778ba97871
paper_authors.avatarUrl: /avatars/fb7c9d4a2c537d918a3267e7cbc03f04.svg
paper_authors.fullname: Xingtai Lv
paper_authors.type: user
paper_authors.user: XingtaiHF
paper_authors._id: 67a19d705efa4fab15497783
paper_authors.name: Shuo Wang
paper_authors._id: 67a19d705efa4fab15497784
paper_authors.name: Yuan Yao
paper_authors._id: 67a19d705efa4fab15497785
paper_authors.name: Xu Han
paper_authors._id: 67a19d705efa4fab15497786
paper_authors.name: Hao Peng
paper_authors._id: 67a19d705efa4fab15497787
paper_authors.name: Yu Cheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-04T14:48:37.956Z
paper_authors._id: 67017abfe4d49b157ac534d9
paper_authors.avatarUrl: /avatars/997e1b9f54b27a7728a9d4abfee4ba91.svg
paper_authors.fullname: Yu Cheng
paper_authors.type: user
paper_authors.user: ych133
paper_authors._id: 67a19d705efa4fab15497788
paper_authors.name: Zhiyuan Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-04T14:53:42.497Z
paper_authors._id: 6310a3cd531cc21f9e06de6a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6310a3cd531cc21f9e06de6a/aTGMx3O41lUARK9s3dAik.jpeg
paper_authors.fullname: Zhiyuan Liu
paper_authors.type: user
paper_authors.user: acharkq
paper_authors._id: 67a19d705efa4fab15497789
paper_authors.name: Maosong Sun
paper_authors._id: 67a19d705efa4fab1549778a
paper_authors.name: Bowen Zhou
paper_authors._id: 67a19d705efa4fab1549778b
paper_authors.name: Ning Ding
paper_title: Process Reinforcement through Implicit Rewards
paper_summary: Dense process rewards have proven a more effective alternative to the sparse
  outcome-level rewards in the inference-time scaling of large language models
  (LLMs), particularly in tasks requiring complex multi-step reasoning. While
  dense rewards also offer an appealing choice for the reinforcement learning
  (RL) of LLMs since their fine-grained rewards have the potential to address
  some inherent issues of outcome rewards, such as training efficiency and credit
  assignment, this potential remains largely unrealized. This can be primarily
  attributed to the challenges of training process reward models (PRMs) online,
  where collecting high-quality process labels is prohibitively expensive, making
  them particularly vulnerable to reward hacking. To address these challenges, we
  propose PRIME (Process Reinforcement through IMplicit rEwards), which enables
  online PRM updates using only policy rollouts and outcome labels through
  implict process rewards. PRIME combines well with various advantage functions
  and forgoes the dedicated reward model training phrase that existing approaches
  require, substantially reducing the development overhead. We demonstrate
  PRIME's effectiveness on competitional math and coding. Starting from
  Qwen2.5-Math-7B-Base, PRIME achieves a 15.1% average improvement across several
  key reasoning benchmarks over the SFT model. Notably, our resulting model,
  Eurus-2-7B-PRIME, surpasses Qwen2.5-Math-7B-Instruct on seven reasoning
  benchmarks with 10% of its training data.
paper_ai_keywords: dense process rewards sparse outcome-level rewards inference-time scaling large language models (LLMs) complex multi-step reasoning reinforcement learning (RL) fine-grained rewards training efficiency credit assignment process reward models (PRMs) reward hacking PRIME (Process Reinforcements through IMplicit rEwards) policy rollouts advantage functions SFT model Qwen2.5-Math-7B-Base reasoning benchmarks Eurus-2-7B-PRIME