Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective
  Reinforcement Learning for LLM Reasoning
Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a
powerful approach to enhancing the reasoning capabilities of Large Language
Models (LLMs), while its mechanisms are not yet well understood. In this work,
we undertake a pioneering exploration of RLVR through the novel perspective of
token entropy patterns, comprehensively analyzing how different tokens
influence reasoning performance. By examining token entropy patterns in
Chain-of-Thought (CoT) reasoning, we observe that only a small fraction of
tokens exhibit high entropy, and these tokens act as critical forks that steer
the model toward diverse reasoning pathways. Furthermore, studying how entropy
patterns evolve during RLVR training reveals that RLVR largely adheres to the
base model's entropy patterns, primarily adjusting the entropy of high-entropy
tokens. These findings highlight the significance of high-entropy tokens (i.e.,
forking tokens) to RLVR. We ultimately improve RLVR by restricting policy
gradient updates to forking tokens and uncover a finding even beyond the 80/20
rule: utilizing only 20% of the tokens while maintaining performance comparable
to full-gradient updates on the Qwen3-8B base model and significantly
surpassing full-gradient updates on the Qwen3-32B (+11.04 on AIME'25 and +7.71
on AIME'24) and Qwen3-14B (+4.79 on AIME'25 and +5.21 on AIME'24) base models,
highlighting a strong scaling trend. In contrast, training exclusively on the
80% lowest-entropy tokens leads to a marked decline in performance. These
findings indicate that the efficacy of RLVR primarily arises from optimizing
the high-entropy tokens that decide reasoning directions. Collectively, our
results highlight the potential to understand RLVR through a token-entropy
perspective and optimize RLVR by leveraging high-entropy minority tokens to
further improve LLM reasoning.
683e7a6d97fd742a8edee1ba
Shenzhi Wang
claimed_verified
2025-06-03T08:41:16.481Z
6486dde1f74857df3f1a5828
https://cdn-avatars.huggingface.co/v1/production/uploads/6486dde1f74857df3f1a5828/FgE80CpalBO5qqArdfwxA.jpeg
Shenzhi Wang
user
shenzhi-wang
683e7a6d97fd742a8edee1bb
Le Yu
claimed_verified
2025-06-04T09:03:00.613Z
6455056fe4952d1c6cb45e49
/avatars/bbdcd4405497f11ff3774352277d1ec7.svg
Le Yu
user
vanillaOVO
683e7a6d97fd742a8edee1bc
Chang Gao
683e7a6d97fd742a8edee1bd
Chujie Zheng
claimed_verified
2025-06-03T08:41:13.605Z
610b70452719facd4ea85e28
https://cdn-avatars.huggingface.co/v1/production/uploads/610b70452719facd4ea85e28/S7nMy7D0Rxq0VIVblhYDG.jpeg
Chujie Zheng
user
chujiezheng
683e7a6d97fd742a8edee1be
Shixuan Liu
claimed_verified
2025-07-29T07:11:11.216Z
64e72776e9fc9d0475ef5188
/avatars/d32b9d4e1da5486c3d5f9b04fa29d167.svg
Shixuan Liu
user
liusx
683e7a6d97fd742a8edee1bf
Rui Lu
683e7a6d97fd742a8edee1c0
Kai Dang
683e7a6d97fd742a8edee1c1
Xionghui Chen
claimed_verified
2025-06-03T08:41:10.358Z
63f30b870a16587ea970edfe
/avatars/059491b33fecec69032e6d481229ee31.svg
Xiong-Hui Chen
user
xionghuichen
683e7a6d97fd742a8edee1c2
Jianxin Yang
683e7a6d97fd742a8edee1c3
Zhenru Zhang
admin_assigned
2025-06-03T09:40:21.027Z
64704e973601bb7b06643e98
/avatars/52e51f4d1be6769e4397b8be2799cf32.svg
Zhenru Zhang
user
Zhenru
683e7a6d97fd742a8edee1c4
Yuqiong Liu
admin_assigned
2025-06-03T09:40:30.240Z
666aacfb918ba11c7c598194
/avatars/45bee8f1fdbdd256ee47d25e4bf01a7a.svg
Yuqiong Liu
user
lyq333
683e7a6d97fd742a8edee1c5
An Yang
claimed_verified
2025-06-09T10:12:45.539Z
62088594a5943c8a8fc94560
https://cdn-avatars.huggingface.co/v1/production/uploads/1644733028938-62088594a5943c8a8fc94560.png
An Yang
user
yangapku
683e7a6d97fd742a8edee1c6
Andrew Zhao
683e7a6d97fd742a8edee1c7
Yang Yue
683e7a6d97fd742a8edee1c8
Shiji Song
683e7a6d97fd742a8edee1c9
Bowen Yu
extracted_pending
2025-06-03T04:30:38.648Z
63d9d68c1cae35c27bf7a6a7
/avatars/b5ad98cf269ae5f1fe90861fb4170fae.svg
Bowen Yu
user
Tigerph
683e7a6d97fd742a8edee1ca
Gao Huang
683e7a6d97fd742a8edee1cb
Junyang Lin
admin_assigned
2025-06-03T09:39:54.278Z
620760a26e3b7210c2ff1943
https://cdn-avatars.huggingface.co/v1/production/uploads/620760a26e3b7210c2ff1943/VC-rKqimF6yxGESNVlPoR.jpeg
Junyang Lin
user
JustinLin610
Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective
  Reinforcement Learning for LLM Reasoning
Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a
powerful approach to enhancing the reasoning capabilities of Large Language
Models (LLMs), while its mechanisms are not yet well understood. In this work,
we undertake a pioneering exploration of RLVR through the novel perspective of
token entropy patterns, comprehensively analyzing how different tokens
influence reasoning performance. By examining token entropy patterns in
Chain-of-Thought (CoT) reasoning, we observe that only a small fraction of
tokens exhibit high entropy, and these tokens act as critical forks that steer
the model toward diverse reasoning pathways. Furthermore, studying how entropy
patterns evolve during RLVR training reveals that RLVR largely adheres to the
base model's entropy patterns, primarily adjusting the entropy of high-entropy
tokens. These findings highlight the significance of high-entropy tokens (i.e.,
forking tokens) to RLVR. We ultimately improve RLVR by restricting policy
gradient updates to forking tokens and uncover a finding even beyond the 80/20
rule: utilizing only 20% of the tokens while maintaining performance comparable
to full-gradient updates on the Qwen3-8B base model and significantly
surpassing full-gradient updates on the Qwen3-32B (+11.04 on AIME'25 and +7.71
on AIME'24) and Qwen3-14B (+4.79 on AIME'25 and +5.21 on AIME'24) base models,
highlighting a strong scaling trend. In contrast, training exclusively on the
80% lowest-entropy tokens leads to a marked decline in performance. These
findings indicate that the efficacy of RLVR primarily arises from optimizing
the high-entropy tokens that decide reasoning directions. Collectively, our
results highlight the potential to understand RLVR through a token-entropy
perspective and optimize RLVR by leveraging high-entropy minority tokens to
further improve LLM reasoning.
Token entropy patterns are crucial in RLVR, with high-entropy tokens significantly impacting reasoning performance and model optimization.
Reinforcement Learning with Verifiable Rewards RLVR Large Language Models LLMs token entropy patterns Chain-of-Thought CoT reasoning high-entropy tokens policy gradient updates Qwen3-8B Qwen3-32B Qwen3-14B AIME