The Entropy Mechanism of Reinforcement Learning for Reasoning Language
  Models
This paper aims to overcome a major obstacle in scaling RL for reasoning with
LLMs, namely the collapse of policy entropy. Such phenomenon is consistently
observed across vast RL runs without entropy intervention, where the policy
entropy dropped sharply at the early training stage, this diminished
exploratory ability is always accompanied with the saturation of policy
performance. In practice, we establish a transformation equation R=-a*e^H+b
between entropy H and downstream performance R. This empirical law strongly
indicates that, the policy performance is traded from policy entropy, thus
bottlenecked by its exhaustion, and the ceiling is fully predictable H=0,
R=-a+b. Our finding necessitates entropy management for continuous exploration
toward scaling compute for RL. To this end, we investigate entropy dynamics
both theoretically and empirically. Our derivation highlights that, the change
in policy entropy is driven by the covariance between action probability and
the change in logits, which is proportional to its advantage when using Policy
Gradient-like algorithms. Empirical study shows that, the values of covariance
term and entropy differences matched exactly, supporting the theoretical
conclusion. Moreover, the covariance term stays mostly positive throughout
training, further explaining why policy entropy would decrease monotonically.
Through understanding the mechanism behind entropy dynamics, we motivate to
control entropy by restricting the update of high-covariance tokens.
Specifically, we propose two simple yet effective techniques, namely Clip-Cov
and KL-Cov, which clip and apply KL penalty to tokens with high covariances
respectively. Experiments show that these methods encourage exploration, thus
helping policy escape entropy collapse and achieve better downstream
performance.
6837cd8fc537d91527323667
Ganqu Cui
claimed_verified
2025-05-29T07:42:09.467Z
650eba9555dc1e841746f132
/avatars/af6f5ee78f161d25ec0afc45d2def8eb.svg
Ganqu Cui
user
ganqu
6837cd8fc537d91527323668
Yuchen Zhang
claimed_verified
2025-05-29T07:42:14.207Z
66e3f8fb5d97b5bb46923444
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/DW806I00-00oQAYvD4ocQ.png
Yuchen Zhang
user
YucZhang2003
6837cd8fc537d91527323669
Jiacheng Chen
claimed_verified
2025-05-29T09:40:20.736Z
65352acb7139c5dd8d9a8590
/avatars/e2ff22b596aee45cdfb8f68dc15572f9.svg
JiachengChen
user
JC-Chen
6837cd8fc537d9152732366a
Lifan Yuan
6837cd8fc537d9152732366b
Zhi Wang
6837cd8fc537d9152732366c
Yuxin Zuo
claimed_verified
2025-05-29T07:42:12.022Z
622474f38dc6b0b64f5e903d
/avatars/d6b60a014277a8ec7d564163c5f644aa.svg
Yuxin Zuo
user
yuxinzuo
6837cd8fc537d9152732366d
Haozhan Li
claimed_verified
2025-05-29T09:40:22.720Z
662f638ba9891e43cc4c5125
/avatars/77c22de5511f9b85d98ec75fb0b5e9be.svg
Li Haozhan
user
Haozhan72
6837cd8fc537d9152732366e
Yuchen Fan
claimed_verified
2025-06-02T07:48:15.931Z
672c2d7816766a76a747b7b5
/avatars/12c7b26d2b81721ccac3a5c71e32a1a1.svg
Yuchen Fan
user
yuchenFan
6837cd8fc537d9152732366f
Huayu Chen
6837cd8fc537d91527323670
Weize Chen
6837cd8fc537d91527323671
Zhiyuan Liu
claimed_verified
2025-06-02T07:48:06.270Z
6472f1f7485a7c8e1bcf507b
/avatars/79dca137583875191ea883b193b630b7.svg
Zhiyuan Liu
user
zibuyu9
6837cd8fc537d91527323672
Hao Peng
6837cd8fc537d91527323673
Lei Bai
6837cd8fc537d91527323674
Wanli Ouyang
6837cd8fc537d91527323675
Yu Cheng
claimed_verified
2025-06-02T07:48:10.357Z
67017abfe4d49b157ac534d9
/avatars/997e1b9f54b27a7728a9d4abfee4ba91.svg
Yu Cheng
user
ych133
6837cd8fc537d91527323676
Bowen Zhou
6837cd8fc537d91527323677
Ning Ding
claimed_verified
2025-05-29T07:42:16.809Z
60cf4bcb1ce3775ebb86e5d5
/avatars/12bcd18d215abf91f297f93007733148.svg
Ning Ding
user
stingning
The Entropy Mechanism of Reinforcement Learning for Reasoning Language
  Models
This paper aims to overcome a major obstacle in scaling RL for reasoning with
LLMs, namely the collapse of policy entropy. Such phenomenon is consistently
observed across vast RL runs without entropy intervention, where the policy
entropy dropped sharply at the early training stage, this diminished
exploratory ability is always accompanied with the saturation of policy
performance. In practice, we establish a transformation equation R=-a*e^H+b
between entropy H and downstream performance R. This empirical law strongly
indicates that, the policy performance is traded from policy entropy, thus
bottlenecked by its exhaustion, and the ceiling is fully predictable H=0,
R=-a+b. Our finding necessitates entropy management for continuous exploration
toward scaling compute for RL. To this end, we investigate entropy dynamics
both theoretically and empirically. Our derivation highlights that, the change
in policy entropy is driven by the covariance between action probability and
the change in logits, which is proportional to its advantage when using Policy
Gradient-like algorithms. Empirical study shows that, the values of covariance
term and entropy differences matched exactly, supporting the theoretical
conclusion. Moreover, the covariance term stays mostly positive throughout
training, further explaining why policy entropy would decrease monotonically.
Through understanding the mechanism behind entropy dynamics, we motivate to
control entropy by restricting the update of high-covariance tokens.
Specifically, we propose two simple yet effective techniques, namely Clip-Cov
and KL-Cov, which clip and apply KL penalty to tokens with high covariances
respectively. Experiments show that these methods encourage exploration, thus
helping policy escape entropy collapse and achieve better downstream
performance.
Entropy dynamics in reinforcement learning with large language models are investigated to prevent policy entropy collapse and improve exploration.
policy entropy reinforcement learning LLMs entropy intervention transformation equation policy performance entropy dynamics covariance action probability logits advantage Policy Gradient Clip-Cov KL-Cov