On-Policy RL with Optimal Reward Baseline
Reinforcement learning algorithms are fundamental to align large language
models with human preferences and to enhance their reasoning capabilities.
However, current reinforcement learning algorithms often suffer from training
instability due to loose on-policy constraints and computational inefficiency
due to auxiliary models. In this work, we propose On-Policy RL with Optimal
reward baseline (OPO), a novel and simplified reinforcement learning algorithm
designed to address these challenges. OPO emphasizes the importance of exact
on-policy training, which empirically stabilizes the training process and
enhances exploration. Moreover, OPO introduces the optimal reward baseline that
theoretically minimizes gradient variance. We evaluate OPO on mathematical
reasoning benchmarks. The results demonstrate its superior performance and
training stability without additional models or regularization terms.
Furthermore, OPO achieves lower policy shifts and higher output entropy,
encouraging more diverse and less repetitive responses. These results highlight
OPO as a promising direction for stable and effective reinforcement learning in
large language model alignment and reasoning tasks. The implementation is
provided at https://github.com/microsoft/LMOps/tree/main/opo.
683916a7a2d6f83cf12552dd
Yaru Hao
683916a7a2d6f83cf12552de
Li Dong
claimed_verified
2025-06-10T09:29:34.742Z
5df85abada6d0311fd3d5408
/avatars/2331cf703c1b5d3a62e2050b1a6eb108.svg
Li Dong
user
unilm
683916a7a2d6f83cf12552df
Xun Wu
683916a7a2d6f83cf12552e0
Shaohan Huang
683916a7a2d6f83cf12552e1
Zewen Chi
683916a7a2d6f83cf12552e2
Furu Wei
On-Policy RL with Optimal Reward Baseline
Reinforcement learning algorithms are fundamental to align large language
models with human preferences and to enhance their reasoning capabilities.
However, current reinforcement learning algorithms often suffer from training
instability due to loose on-policy constraints and computational inefficiency
due to auxiliary models. In this work, we propose On-Policy RL with Optimal
reward baseline (OPO), a novel and simplified reinforcement learning algorithm
designed to address these challenges. OPO emphasizes the importance of exact
on-policy training, which empirically stabilizes the training process and
enhances exploration. Moreover, OPO introduces the optimal reward baseline that
theoretically minimizes gradient variance. We evaluate OPO on mathematical
reasoning benchmarks. The results demonstrate its superior performance and
training stability without additional models or regularization terms.
Furthermore, OPO achieves lower policy shifts and higher output entropy,
encouraging more diverse and less repetitive responses. These results highlight
OPO as a promising direction for stable and effective reinforcement learning in
large language model alignment and reasoning tasks. The implementation is
provided at https://github.com/microsoft/LMOps/tree/main/opo.
A novel reinforcement learning algorithm, OPO, improves training stability and performance in large language model alignment and reasoning by emphasizing exact on-policy training and using an optimal reward baseline.
reinforcement learning on-policy constraints computational efficiency auxiliary models optimal reward baseline gradient variance policy shifts output entropy mathematical reasoning benchmarks