title: CPGD: Toward Stable Rule-based Reinforcement Learning for Language
    Models
summary: Recent advances in rule-based reinforcement learning (RL) have significantly
  improved the reasoning capability of language models (LMs) with rule-based
  rewards. However, existing RL methods -- such as GRPO, REINFORCE++, and RLOO --
  often suffer from training instability, where large policy updates and improper
  clipping can lead to training collapse. To address this issue, we propose
  Clipped Policy Gradient Optimization with Policy Drift (CPGD), a novel
  algorithm designed to stabilize policy learning in LMs. CPGD introduces a
  policy drift constraint based on KL divergence to dynamically regularize policy
  updates, and leverages a clip mechanism on the logarithm of the ratio to
  prevent excessive policy updates. We provide theoretical justification for CPGD
  and demonstrate through empirical analysis that it mitigates the instability
  observed in prior approaches. Furthermore, we show that CPGD significantly
  improves performance while maintaining training stability. Our implementation
  balances theoretical rigor with practical usability, offering a robust
  alternative for RL in the post-training of LMs. We release our code at
  https://github.com/ModalMinds/MM-EUREKA.
paper_authors._id: 682bf9090080c5ce0c1b43a1
paper_authors.name: Zongkai Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-20T08:32:24.351Z
paper_authors._id: 674d42a03a4b7e31a1707218
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/3DIez-RYnDMYe1U-m0qBZ.png
paper_authors.fullname: kkkai
paper_authors.type: user
paper_authors.user: Zkkkai
paper_authors._id: 682bf9090080c5ce0c1b43a2
paper_authors.name: Fanqing Meng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-20T08:32:31.174Z
paper_authors._id: 640b37b2bab5ca8fbe7df8f2
paper_authors.avatarUrl: /avatars/c7bef45efad6a0d911a720e2236fcba5.svg
paper_authors.fullname: fanqing meng
paper_authors.type: user
paper_authors.user: FanqingM
paper_authors._id: 682bf9090080c5ce0c1b43a3
paper_authors.name: Lingxiao Du
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-20T08:32:46.648Z
paper_authors._id: 666fe1a5b07525f0bde69c27
paper_authors.avatarUrl: /avatars/bb98ab0b974c8fe011739baa8dadd91a.svg
paper_authors.fullname: Lingxiao Du
paper_authors.type: user
paper_authors.user: Cierra0506
paper_authors._id: 682bf9090080c5ce0c1b43a4
paper_authors.name: Zhixiang Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-20T08:32:54.272Z
paper_authors._id: 674bfdf227f531cdc248bb5c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/674bfdf227f531cdc248bb5c/xh4gw89sr8MzNzRdiTjFx.jpeg
paper_authors.fullname: Zhixiang Zhou
paper_authors.type: user
paper_authors.user: SuperposedWave
paper_authors._id: 682bf9090080c5ce0c1b43a5
paper_authors.name: Chao Yu
paper_authors._id: 682bf9090080c5ce0c1b43a6
paper_authors.name: Wenqi Shao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-20T08:33:04.198Z
paper_authors._id: 64b3fd42eec33e27dcc4c941
paper_authors.avatarUrl: /avatars/5aa1a99468fa61d4b8b0e80b592c4e55.svg
paper_authors.fullname: Wenqi Shao
paper_authors.type: user
paper_authors.user: wqshao126
paper_authors._id: 682bf9090080c5ce0c1b43a7
paper_authors.name: Qiaosheng Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-20T08:33:09.646Z
paper_authors._id: 63cf4ecdc1dedf59c8f8362e
paper_authors.avatarUrl: /avatars/cede885854d6a1551860080d55c87568.svg
paper_authors.fullname: Qiaosheng ZHANG
paper_authors.type: user
paper_authors.user: Domingo12
paper_title: CPGD: Toward Stable Rule-based Reinforcement Learning for Language
    Models
paper_summary: Recent advances in rule-based reinforcement learning (RL) have significantly
  improved the reasoning capability of language models (LMs) with rule-based
  rewards. However, existing RL methods -- such as GRPO, REINFORCE++, and RLOO --
  often suffer from training instability, where large policy updates and improper
  clipping can lead to training collapse. To address this issue, we propose
  Clipped Policy Gradient Optimization with Policy Drift (CPGD), a novel
  algorithm designed to stabilize policy learning in LMs. CPGD introduces a
  policy drift constraint based on KL divergence to dynamically regularize policy
  updates, and leverages a clip mechanism on the logarithm of the ratio to
  prevent excessive policy updates. We provide theoretical justification for CPGD
  and demonstrate through empirical analysis that it mitigates the instability
  observed in prior approaches. Furthermore, we show that CPGD significantly
  improves performance while maintaining training stability. Our implementation
  balances theoretical rigor with practical usability, offering a robust
  alternative for RL in the post-training of LMs. We release our code at
  https://github.com/ModalMinds/MM-EUREKA.
paper_ai_summary: A novel reinforcement learning algorithm, CPGD, stabilizes policy learning in language models by constraining policy drift and clipping updates, improving performance and stability.
paper_ai_keywords: rule-based reinforcement learning reinforcement learning language models GRPO REINFORCE++ RLOO Clipped Policy Gradient Optimization with Policy Drift CPGD policy drift constraint KL divergence policy updates training instability