title: The Entropy Mechanism of Reinforcement Learning for Reasoning Language
    Models
summary: This paper aims to overcome a major obstacle in scaling RL for reasoning with
  LLMs, namely the collapse of policy entropy. Such phenomenon is consistently
  observed across vast RL runs without entropy intervention, where the policy
  entropy dropped sharply at the early training stage, this diminished
  exploratory ability is always accompanied with the saturation of policy
  performance. In practice, we establish a transformation equation R=-a*e^H+b
  between entropy H and downstream performance R. This empirical law strongly
  indicates that, the policy performance is traded from policy entropy, thus
  bottlenecked by its exhaustion, and the ceiling is fully predictable H=0,
  R=-a+b. Our finding necessitates entropy management for continuous exploration
  toward scaling compute for RL. To this end, we investigate entropy dynamics
  both theoretically and empirically. Our derivation highlights that, the change
  in policy entropy is driven by the covariance between action probability and
  the change in logits, which is proportional to its advantage when using Policy
  Gradient-like algorithms. Empirical study shows that, the values of covariance
  term and entropy differences matched exactly, supporting the theoretical
  conclusion. Moreover, the covariance term stays mostly positive throughout
  training, further explaining why policy entropy would decrease monotonically.
  Through understanding the mechanism behind entropy dynamics, we motivate to
  control entropy by restricting the update of high-covariance tokens.
  Specifically, we propose two simple yet effective techniques, namely Clip-Cov
  and KL-Cov, which clip and apply KL penalty to tokens with high covariances
  respectively. Experiments show that these methods encourage exploration, thus
  helping policy escape entropy collapse and achieve better downstream
  performance.
paper_authors._id: 6837cd8fc537d91527323667
paper_authors.name: Ganqu Cui
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-29T07:42:09.467Z
paper_authors._id: 650eba9555dc1e841746f132
paper_authors.avatarUrl: /avatars/af6f5ee78f161d25ec0afc45d2def8eb.svg
paper_authors.fullname: Ganqu Cui
paper_authors.type: user
paper_authors.user: ganqu
paper_authors._id: 6837cd8fc537d91527323668
paper_authors.name: Yuchen Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-29T07:42:14.207Z
paper_authors._id: 66e3f8fb5d97b5bb46923444
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/DW806I00-00oQAYvD4ocQ.png
paper_authors.fullname: Yuchen Zhang
paper_authors.type: user
paper_authors.user: YucZhang2003
paper_authors._id: 6837cd8fc537d91527323669
paper_authors.name: Jiacheng Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-29T09:40:20.736Z
paper_authors._id: 65352acb7139c5dd8d9a8590
paper_authors.avatarUrl: /avatars/e2ff22b596aee45cdfb8f68dc15572f9.svg
paper_authors.fullname: JiachengChen
paper_authors.type: user
paper_authors.user: JC-Chen
paper_authors._id: 6837cd8fc537d9152732366a
paper_authors.name: Lifan Yuan
paper_authors._id: 6837cd8fc537d9152732366b
paper_authors.name: Zhi Wang
paper_authors._id: 6837cd8fc537d9152732366c
paper_authors.name: Yuxin Zuo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-29T07:42:12.022Z
paper_authors._id: 622474f38dc6b0b64f5e903d
paper_authors.avatarUrl: /avatars/d6b60a014277a8ec7d564163c5f644aa.svg
paper_authors.fullname: Yuxin Zuo
paper_authors.type: user
paper_authors.user: yuxinzuo
paper_authors._id: 6837cd8fc537d9152732366d
paper_authors.name: Haozhan Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-29T09:40:22.720Z
paper_authors._id: 662f638ba9891e43cc4c5125
paper_authors.avatarUrl: /avatars/77c22de5511f9b85d98ec75fb0b5e9be.svg
paper_authors.fullname: Li Haozhan
paper_authors.type: user
paper_authors.user: Haozhan72
paper_authors._id: 6837cd8fc537d9152732366e
paper_authors.name: Yuchen Fan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-02T07:48:15.931Z
paper_authors._id: 672c2d7816766a76a747b7b5
paper_authors.avatarUrl: /avatars/12c7b26d2b81721ccac3a5c71e32a1a1.svg
paper_authors.fullname: Yuchen Fan
paper_authors.type: user
paper_authors.user: yuchenFan
paper_authors._id: 6837cd8fc537d9152732366f
paper_authors.name: Huayu Chen
paper_authors._id: 6837cd8fc537d91527323670
paper_authors.name: Weize Chen
paper_authors._id: 6837cd8fc537d91527323671
paper_authors.name: Zhiyuan Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-02T07:48:06.270Z
paper_authors._id: 6472f1f7485a7c8e1bcf507b
paper_authors.avatarUrl: /avatars/79dca137583875191ea883b193b630b7.svg
paper_authors.fullname: Zhiyuan Liu
paper_authors.type: user
paper_authors.user: zibuyu9
paper_authors._id: 6837cd8fc537d91527323672
paper_authors.name: Hao Peng
paper_authors._id: 6837cd8fc537d91527323673
paper_authors.name: Lei Bai
paper_authors._id: 6837cd8fc537d91527323674
paper_authors.name: Wanli Ouyang
paper_authors._id: 6837cd8fc537d91527323675
paper_authors.name: Yu Cheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-02T07:48:10.357Z
paper_authors._id: 67017abfe4d49b157ac534d9
paper_authors.avatarUrl: /avatars/997e1b9f54b27a7728a9d4abfee4ba91.svg
paper_authors.fullname: Yu Cheng
paper_authors.type: user
paper_authors.user: ych133
paper_authors._id: 6837cd8fc537d91527323676
paper_authors.name: Bowen Zhou
paper_authors._id: 6837cd8fc537d91527323677
paper_authors.name: Ning Ding
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-29T07:42:16.809Z
paper_authors._id: 60cf4bcb1ce3775ebb86e5d5
paper_authors.avatarUrl: /avatars/12bcd18d215abf91f297f93007733148.svg
paper_authors.fullname: Ning Ding
paper_authors.type: user
paper_authors.user: stingning
paper_title: The Entropy Mechanism of Reinforcement Learning for Reasoning Language
    Models
paper_summary: This paper aims to overcome a major obstacle in scaling RL for reasoning with
  LLMs, namely the collapse of policy entropy. Such phenomenon is consistently
  observed across vast RL runs without entropy intervention, where the policy
  entropy dropped sharply at the early training stage, this diminished
  exploratory ability is always accompanied with the saturation of policy
  performance. In practice, we establish a transformation equation R=-a*e^H+b
  between entropy H and downstream performance R. This empirical law strongly
  indicates that, the policy performance is traded from policy entropy, thus
  bottlenecked by its exhaustion, and the ceiling is fully predictable H=0,
  R=-a+b. Our finding necessitates entropy management for continuous exploration
  toward scaling compute for RL. To this end, we investigate entropy dynamics
  both theoretically and empirically. Our derivation highlights that, the change
  in policy entropy is driven by the covariance between action probability and
  the change in logits, which is proportional to its advantage when using Policy
  Gradient-like algorithms. Empirical study shows that, the values of covariance
  term and entropy differences matched exactly, supporting the theoretical
  conclusion. Moreover, the covariance term stays mostly positive throughout
  training, further explaining why policy entropy would decrease monotonically.
  Through understanding the mechanism behind entropy dynamics, we motivate to
  control entropy by restricting the update of high-covariance tokens.
  Specifically, we propose two simple yet effective techniques, namely Clip-Cov
  and KL-Cov, which clip and apply KL penalty to tokens with high covariances
  respectively. Experiments show that these methods encourage exploration, thus
  helping policy escape entropy collapse and achieve better downstream
  performance.
paper_ai_summary: Entropy dynamics in reinforcement learning with large language models are investigated to prevent policy entropy collapse and improve exploration.
paper_ai_keywords: policy entropy reinforcement learning LLMs entropy intervention transformation equation policy performance entropy dynamics covariance action probability logits advantage Policy Gradient Clip-Cov KL-Cov