title: On-Policy RL with Optimal Reward Baseline
summary: Reinforcement learning algorithms are fundamental to align large language
  models with human preferences and to enhance their reasoning capabilities.
  However, current reinforcement learning algorithms often suffer from training
  instability due to loose on-policy constraints and computational inefficiency
  due to auxiliary models. In this work, we propose On-Policy RL with Optimal
  reward baseline (OPO), a novel and simplified reinforcement learning algorithm
  designed to address these challenges. OPO emphasizes the importance of exact
  on-policy training, which empirically stabilizes the training process and
  enhances exploration. Moreover, OPO introduces the optimal reward baseline that
  theoretically minimizes gradient variance. We evaluate OPO on mathematical
  reasoning benchmarks. The results demonstrate its superior performance and
  training stability without additional models or regularization terms.
  Furthermore, OPO achieves lower policy shifts and higher output entropy,
  encouraging more diverse and less repetitive responses. These results highlight
  OPO as a promising direction for stable and effective reinforcement learning in
  large language model alignment and reasoning tasks. The implementation is
  provided at https://github.com/microsoft/LMOps/tree/main/opo.
paper_authors._id: 683916a7a2d6f83cf12552dd
paper_authors.name: Yaru Hao
paper_authors._id: 683916a7a2d6f83cf12552de
paper_authors.name: Li Dong
paper_authors._id: 683916a7a2d6f83cf12552df
paper_authors.name: Xun Wu
paper_authors._id: 683916a7a2d6f83cf12552e0
paper_authors.name: Shaohan Huang
paper_authors._id: 683916a7a2d6f83cf12552e1
paper_authors.name: Zewen Chi
paper_authors._id: 683916a7a2d6f83cf12552e2
paper_authors.name: Furu Wei
paper_title: On-Policy RL with Optimal Reward Baseline
paper_summary: Reinforcement learning algorithms are fundamental to align large language
  models with human preferences and to enhance their reasoning capabilities.
  However, current reinforcement learning algorithms often suffer from training
  instability due to loose on-policy constraints and computational inefficiency
  due to auxiliary models. In this work, we propose On-Policy RL with Optimal
  reward baseline (OPO), a novel and simplified reinforcement learning algorithm
  designed to address these challenges. OPO emphasizes the importance of exact
  on-policy training, which empirically stabilizes the training process and
  enhances exploration. Moreover, OPO introduces the optimal reward baseline that
  theoretically minimizes gradient variance. We evaluate OPO on mathematical
  reasoning benchmarks. The results demonstrate its superior performance and
  training stability without additional models or regularization terms.
  Furthermore, OPO achieves lower policy shifts and higher output entropy,
  encouraging more diverse and less repetitive responses. These results highlight
  OPO as a promising direction for stable and effective reinforcement learning in
  large language model alignment and reasoning tasks. The implementation is
  provided at https://github.com/microsoft/LMOps/tree/main/opo.
paper_ai_summary: A novel reinforcement learning algorithm, OPO, improves training stability and performance in large language model alignment and reasoning by emphasizing exact on-policy training and using an optimal reward baseline.
paper_ai_keywords: reinforcement learning on-policy constraints computational efficiency auxiliary models optimal reward baseline gradient variance policy shifts output entropy mathematical reasoning benchmarks