Process Reinforcement through Implicit Rewards
Dense process rewards have proven a more effective alternative to the sparse
outcome-level rewards in the inference-time scaling of large language models
(LLMs), particularly in tasks requiring complex multi-step reasoning. While
dense rewards also offer an appealing choice for the reinforcement learning
(RL) of LLMs since their fine-grained rewards have the potential to address
some inherent issues of outcome rewards, such as training efficiency and credit
assignment, this potential remains largely unrealized. This can be primarily
attributed to the challenges of training process reward models (PRMs) online,
where collecting high-quality process labels is prohibitively expensive, making
them particularly vulnerable to reward hacking. To address these challenges, we
propose PRIME (Process Reinforcement through IMplicit rEwards), which enables
online PRM updates using only policy rollouts and outcome labels through
implict process rewards. PRIME combines well with various advantage functions
and forgoes the dedicated reward model training phrase that existing approaches
require, substantially reducing the development overhead. We demonstrate
PRIME's effectiveness on competitional math and coding. Starting from
Qwen2.5-Math-7B-Base, PRIME achieves a 15.1% average improvement across several
key reasoning benchmarks over the SFT model. Notably, our resulting model,
Eurus-2-7B-PRIME, surpasses Qwen2.5-Math-7B-Instruct on seven reasoning
benchmarks with 10% of its training data.
67a19d705efa4fab15497775
Ganqu Cui
claimed_verified
2025-02-04T09:39:23.889Z
650eba9555dc1e841746f132
/avatars/af6f5ee78f161d25ec0afc45d2def8eb.svg
Ganqu Cui
user
ganqu
67a19d705efa4fab15497776
Lifan Yuan
67a19d705efa4fab15497777
Zefan Wang
67a19d705efa4fab15497778
Hanbin Wang
claimed_verified
2025-02-04T09:39:25.869Z
6321152b8c0da827c72c7c16
https://cdn-avatars.huggingface.co/v1/production/uploads/1678783813705-6321152b8c0da827c72c7c16.jpeg
Hanbin Wang
user
hanbin
67a19d705efa4fab15497779
Wendi Li
admin_assigned
2025-02-04T14:51:24.261Z
671bfaa29e5e675c7f5c4307
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/PwDA6OSSAmg6k4LliEQkZ.png
Wendi Li
user
wendili
67a19d705efa4fab1549777a
Bingxiang He
admin_assigned
2025-02-04T14:51:31.090Z
64c5e944979493279b700cb2
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/vjFuPWw8Vl7b7gXB19Sk-.jpeg
Bingxiang He
user
hbx
67a19d705efa4fab1549777b
Yuchen Fan
admin_assigned
2025-02-04T14:51:56.597Z
672c2d7816766a76a747b7b5
/avatars/12c7b26d2b81721ccac3a5c71e32a1a1.svg
Yuchen Fan
user
yuchenFan
67a19d705efa4fab1549777c
Tianyu Yu
admin_assigned
2025-02-04T14:52:26.615Z
64abc4aa6cadc7aca585dddf
/avatars/736afea979cd0021c7a37f68731524ea.svg
Tianyu Yu
user
Yirany
67a19d705efa4fab1549777d
Qixin Xu
claimed_verified
2025-02-06T14:15:22.453Z
6680f0b20b72be136708af26
/avatars/5d8fd5be0cf94e246b46abb9d3cc8f5c.svg
XuQixin
user
Racktic
67a19d705efa4fab1549777e
Weize Chen
admin_assigned
2025-02-04T14:52:46.343Z
648312243b7fe59c876c0dca
/avatars/c26ad76cd213529e4670bb599b8199bb.svg
weize
user
weizechen
67a19d705efa4fab1549777f
Jiarui Yuan
67a19d705efa4fab15497780
Huayu Chen
admin_assigned
2025-02-04T14:53:06.620Z
6630f87ee53fcb71c3887df0
/avatars/50191a3d45bebf90cf08df09477e95db.svg
HuayuChen
user
HuayuChen
67a19d705efa4fab15497781
Kaiyan Zhang
claimed_verified
2025-02-11T15:36:03.829Z
60bc94cd85a3ab33829b6211
/avatars/b57d36c7577fbbb42ea5b963eef4144a.svg
Kaiyan Zhang
user
iseesaw
67a19d705efa4fab15497782
Xingtai Lv
admin_assigned
2025-02-04T14:53:21.172Z
663f07d029be04778ba97871
/avatars/fb7c9d4a2c537d918a3267e7cbc03f04.svg
Xingtai Lv
user
XingtaiHF
67a19d705efa4fab15497783
Shuo Wang
67a19d705efa4fab15497784
Yuan Yao
67a19d705efa4fab15497785
Xu Han
67a19d705efa4fab15497786
Hao Peng
67a19d705efa4fab15497787
Yu Cheng
claimed_verified
2025-02-04T14:48:37.956Z
67017abfe4d49b157ac534d9
/avatars/997e1b9f54b27a7728a9d4abfee4ba91.svg
Yu Cheng
user
ych133
67a19d705efa4fab15497788
Zhiyuan Liu
admin_assigned
2025-02-04T14:53:42.497Z
6310a3cd531cc21f9e06de6a
https://cdn-avatars.huggingface.co/v1/production/uploads/6310a3cd531cc21f9e06de6a/aTGMx3O41lUARK9s3dAik.jpeg
Zhiyuan Liu
user
acharkq
67a19d705efa4fab15497789
Maosong Sun
67a19d705efa4fab1549778a
Bowen Zhou
67a19d705efa4fab1549778b
Ning Ding
Process Reinforcement through Implicit Rewards
Dense process rewards have proven a more effective alternative to the sparse
outcome-level rewards in the inference-time scaling of large language models
(LLMs), particularly in tasks requiring complex multi-step reasoning. While
dense rewards also offer an appealing choice for the reinforcement learning
(RL) of LLMs since their fine-grained rewards have the potential to address
some inherent issues of outcome rewards, such as training efficiency and credit
assignment, this potential remains largely unrealized. This can be primarily
attributed to the challenges of training process reward models (PRMs) online,
where collecting high-quality process labels is prohibitively expensive, making
them particularly vulnerable to reward hacking. To address these challenges, we
propose PRIME (Process Reinforcement through IMplicit rEwards), which enables
online PRM updates using only policy rollouts and outcome labels through
implict process rewards. PRIME combines well with various advantage functions
and forgoes the dedicated reward model training phrase that existing approaches
require, substantially reducing the development overhead. We demonstrate
PRIME's effectiveness on competitional math and coding. Starting from
Qwen2.5-Math-7B-Base, PRIME achieves a 15.1% average improvement across several
key reasoning benchmarks over the SFT model. Notably, our resulting model,
Eurus-2-7B-PRIME, surpasses Qwen2.5-Math-7B-Instruct on seven reasoning
benchmarks with 10% of its training data.
PRIME leverages implicit process rewards to improve the reinforcement learning of large language models, achieving better performance with less data compared to traditional methods.
dense process rewards sparse outcome-level rewards large language models reinforcement learning process reward models reward hacking policy rollouts advantage functions Qwen2.5-Math-7B-Base reasoning benchmarks Eurus-2-7B-PRIME