On the Design of KL-Regularized Policy Gradient Algorithms for LLM
  Reasoning
Policy gradient algorithms have been successfully applied to enhance the
reasoning capabilities of large language models (LLMs). Despite the widespread
use of Kullback-Leibler (KL) regularization in policy gradient algorithms to
stabilize training, the systematic exploration of how different KL divergence
formulations can be estimated and integrated into surrogate loss functions for
online reinforcement learning (RL) presents a nuanced and systematically
explorable design space. In this paper, we propose regularized policy gradient
(RPG), a systematic framework for deriving and analyzing KL-regularized policy
gradient methods in the online RL setting. We derive policy gradients and
corresponding surrogate loss functions for objectives regularized by both
forward and reverse KL divergences, considering both normalized and
unnormalized policy distributions. Furthermore, we present derivations for
fully differentiable loss functions as well as REINFORCE-style gradient
estimators, accommodating diverse algorithmic needs. We conduct extensive
experiments on RL for LLM reasoning using these methods, showing improved or
competitive results in terms of training stability and performance compared to
strong baselines such as GRPO, REINFORCE++, and DAPO. The code is available at
https://github.com/complex-reasoning/RPG.
6833cf5a2d728e2330d572e3
Yifan Zhang
claimed_verified
2025-05-26T08:10:09.930Z
647bf082aba7062fe5c51ca9
https://cdn-avatars.huggingface.co/v1/production/uploads/647bf082aba7062fe5c51ca9/p4lY9IjHiWZETKmFq1mtU.jpeg
Yifan Zhang
user
yifAI
6833cf5a2d728e2330d572e4
Yifeng Liu
claimed_verified
2025-05-26T08:10:06.913Z
653d276681f52ceb4d12bd85
/avatars/56601a25e5f883a8f6dc15f6fd9dcc57.svg
Yifeng Liu
user
Lewis-Lau
6833cf5a2d728e2330d572e5
Huizhuo Yuan
6833cf5a2d728e2330d572e6
Yang Yuan
6833cf5a2d728e2330d572e7
Quanquan Gu
6833cf5a2d728e2330d572e8
Andrew C Yao
On the Design of KL-Regularized Policy Gradient Algorithms for LLM
  Reasoning
Policy gradient algorithms have been successfully applied to enhance the
reasoning capabilities of large language models (LLMs). Despite the widespread
use of Kullback-Leibler (KL) regularization in policy gradient algorithms to
stabilize training, the systematic exploration of how different KL divergence
formulations can be estimated and integrated into surrogate loss functions for
online reinforcement learning (RL) presents a nuanced and systematically
explorable design space. In this paper, we propose regularized policy gradient
(RPG), a systematic framework for deriving and analyzing KL-regularized policy
gradient methods in the online RL setting. We derive policy gradients and
corresponding surrogate loss functions for objectives regularized by both
forward and reverse KL divergences, considering both normalized and
unnormalized policy distributions. Furthermore, we present derivations for
fully differentiable loss functions as well as REINFORCE-style gradient
estimators, accommodating diverse algorithmic needs. We conduct extensive
experiments on RL for LLM reasoning using these methods, showing improved or
competitive results in terms of training stability and performance compared to
strong baselines such as GRPO, REINFORCE++, and DAPO. The code is available at
https://github.com/complex-reasoning/RPG.
A regularized policy gradient framework is introduced to explore KL divergence formulations for enhancing the reasoning capabilities of LLMs in online reinforcement learning, demonstrating improved training stability and performance.
policy gradient algorithms KL regularization KL divergence surrogate loss functions online reinforcement learning full differentiable loss functions REINFORCE-style gradient estimators GRPO REINFORCE++ DAPO regularized policy gradient (RPG) forward KL divergence reverse KL divergence normalized policy distributions unnormalized policy distributions