title: On the Design of KL-Regularized Policy Gradient Algorithms for LLM
    Reasoning
summary: Policy gradient algorithms have been successfully applied to enhance the
  reasoning capabilities of large language models (LLMs). Despite the widespread
  use of Kullback-Leibler (KL) regularization in policy gradient algorithms to
  stabilize training, the systematic exploration of how different KL divergence
  formulations can be estimated and integrated into surrogate loss functions for
  online reinforcement learning (RL) presents a nuanced and systematically
  explorable design space. In this paper, we propose regularized policy gradient
  (RPG), a systematic framework for deriving and analyzing KL-regularized policy
  gradient methods in the online RL setting. We derive policy gradients and
  corresponding surrogate loss functions for objectives regularized by both
  forward and reverse KL divergences, considering both normalized and
  unnormalized policy distributions. Furthermore, we present derivations for
  fully differentiable loss functions as well as REINFORCE-style gradient
  estimators, accommodating diverse algorithmic needs. We conduct extensive
  experiments on RL for LLM reasoning using these methods, showing improved or
  competitive results in terms of training stability and performance compared to
  strong baselines such as GRPO, REINFORCE++, and DAPO. The code is available at
  https://github.com/complex-reasoning/RPG.
paper_authors._id: 6833cf5a2d728e2330d572e3
paper_authors.name: Yifan Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-26T08:10:09.930Z
paper_authors._id: 647bf082aba7062fe5c51ca9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/647bf082aba7062fe5c51ca9/p4lY9IjHiWZETKmFq1mtU.jpeg
paper_authors.fullname: Yifan Zhang
paper_authors.type: user
paper_authors.user: yifAI
paper_authors._id: 6833cf5a2d728e2330d572e4
paper_authors.name: Yifeng Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-26T08:10:06.913Z
paper_authors._id: 653d276681f52ceb4d12bd85
paper_authors.avatarUrl: /avatars/56601a25e5f883a8f6dc15f6fd9dcc57.svg
paper_authors.fullname: Yifeng Liu
paper_authors.type: user
paper_authors.user: Lewis-Lau
paper_authors._id: 6833cf5a2d728e2330d572e5
paper_authors.name: Huizhuo Yuan
paper_authors._id: 6833cf5a2d728e2330d572e6
paper_authors.name: Yang Yuan
paper_authors._id: 6833cf5a2d728e2330d572e7
paper_authors.name: Quanquan Gu
paper_authors._id: 6833cf5a2d728e2330d572e8
paper_authors.name: Andrew C Yao
paper_title: On the Design of KL-Regularized Policy Gradient Algorithms for LLM
    Reasoning
paper_summary: Policy gradient algorithms have been successfully applied to enhance the
  reasoning capabilities of large language models (LLMs). Despite the widespread
  use of Kullback-Leibler (KL) regularization in policy gradient algorithms to
  stabilize training, the systematic exploration of how different KL divergence
  formulations can be estimated and integrated into surrogate loss functions for
  online reinforcement learning (RL) presents a nuanced and systematically
  explorable design space. In this paper, we propose regularized policy gradient
  (RPG), a systematic framework for deriving and analyzing KL-regularized policy
  gradient methods in the online RL setting. We derive policy gradients and
  corresponding surrogate loss functions for objectives regularized by both
  forward and reverse KL divergences, considering both normalized and
  unnormalized policy distributions. Furthermore, we present derivations for
  fully differentiable loss functions as well as REINFORCE-style gradient
  estimators, accommodating diverse algorithmic needs. We conduct extensive
  experiments on RL for LLM reasoning using these methods, showing improved or
  competitive results in terms of training stability and performance compared to
  strong baselines such as GRPO, REINFORCE++, and DAPO. The code is available at
  https://github.com/complex-reasoning/RPG.
paper_ai_summary: A regularized policy gradient framework is introduced to explore KL divergence formulations for enhancing the reasoning capabilities of LLMs in online reinforcement learning, demonstrating improved training stability and performance.
paper_ai_keywords: policy gradient algorithms KL regularization KL divergence surrogate loss functions online reinforcement learning full differentiable loss functions REINFORCE-style gradient estimators GRPO REINFORCE++ DAPO regularized policy gradient (RPG) forward KL divergence reverse KL divergence normalized policy distributions unnormalized policy distributions