title: Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective
    Reinforcement Learning for LLM Reasoning
summary: Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a
  powerful approach to enhancing the reasoning capabilities of Large Language
  Models (LLMs), while its mechanisms are not yet well understood. In this work,
  we undertake a pioneering exploration of RLVR through the novel perspective of
  token entropy patterns, comprehensively analyzing how different tokens
  influence reasoning performance. By examining token entropy patterns in
  Chain-of-Thought (CoT) reasoning, we observe that only a small fraction of
  tokens exhibit high entropy, and these tokens act as critical forks that steer
  the model toward diverse reasoning pathways. Furthermore, studying how entropy
  patterns evolve during RLVR training reveals that RLVR largely adheres to the
  base model's entropy patterns, primarily adjusting the entropy of high-entropy
  tokens. These findings highlight the significance of high-entropy tokens (i.e.,
  forking tokens) to RLVR. We ultimately improve RLVR by restricting policy
  gradient updates to forking tokens and uncover a finding even beyond the 80/20
  rule: utilizing only 20% of the tokens while maintaining performance comparable
  to full-gradient updates on the Qwen3-8B base model and significantly
  surpassing full-gradient updates on the Qwen3-32B (+11.04 on AIME'25 and +7.71
  on AIME'24) and Qwen3-14B (+4.79 on AIME'25 and +5.21 on AIME'24) base models,
  highlighting a strong scaling trend. In contrast, training exclusively on the
  80% lowest-entropy tokens leads to a marked decline in performance. These
  findings indicate that the efficacy of RLVR primarily arises from optimizing
  the high-entropy tokens that decide reasoning directions. Collectively, our
  results highlight the potential to understand RLVR through a token-entropy
  perspective and optimize RLVR by leveraging high-entropy minority tokens to
  further improve LLM reasoning.
paper_authors._id: 683e7a6d97fd742a8edee1ba
paper_authors.name: Shenzhi Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-03T08:41:16.481Z
paper_authors._id: 6486dde1f74857df3f1a5828
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6486dde1f74857df3f1a5828/FgE80CpalBO5qqArdfwxA.jpeg
paper_authors.fullname: Shenzhi Wang
paper_authors.type: user
paper_authors.user: shenzhi-wang
paper_authors._id: 683e7a6d97fd742a8edee1bb
paper_authors.name: Le Yu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-04T09:03:00.613Z
paper_authors._id: 6455056fe4952d1c6cb45e49
paper_authors.avatarUrl: /avatars/730f61ebc4f2df7489499ad7471a426e.svg
paper_authors.fullname: Le Yu
paper_authors.type: user
paper_authors.user: vanillaOVO
paper_authors._id: 683e7a6d97fd742a8edee1bc
paper_authors.name: Chang Gao
paper_authors._id: 683e7a6d97fd742a8edee1bd
paper_authors.name: Chujie Zheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-03T08:41:13.605Z
paper_authors._id: 610b70452719facd4ea85e28
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/610b70452719facd4ea85e28/S7nMy7D0Rxq0VIVblhYDG.jpeg
paper_authors.fullname: Chujie Zheng
paper_authors.type: user
paper_authors.user: chujiezheng
paper_authors._id: 683e7a6d97fd742a8edee1be
paper_authors.name: Shixuan Liu
paper_authors._id: 683e7a6d97fd742a8edee1bf
paper_authors.name: Rui Lu
paper_authors._id: 683e7a6d97fd742a8edee1c0
paper_authors.name: Kai Dang
paper_authors._id: 683e7a6d97fd742a8edee1c1
paper_authors.name: Xionghui Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-03T08:41:10.358Z
paper_authors._id: 63f30b870a16587ea970edfe
paper_authors.avatarUrl: /avatars/059491b33fecec69032e6d481229ee31.svg
paper_authors.fullname: Xiong-Hui Chen
paper_authors.type: user
paper_authors.user: xionghuichen
paper_authors._id: 683e7a6d97fd742a8edee1c2
paper_authors.name: Jianxin Yang
paper_authors._id: 683e7a6d97fd742a8edee1c3
paper_authors.name: Zhenru Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-06-03T09:40:21.027Z
paper_authors._id: 64704e973601bb7b06643e98
paper_authors.avatarUrl: /avatars/52e51f4d1be6769e4397b8be2799cf32.svg
paper_authors.fullname: Zhenru Zhang
paper_authors.type: user
paper_authors.user: Zhenru
paper_authors._id: 683e7a6d97fd742a8edee1c4
paper_authors.name: Yuqiong Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-06-03T09:40:30.240Z
paper_authors._id: 666aacfb918ba11c7c598194
paper_authors.avatarUrl: /avatars/45bee8f1fdbdd256ee47d25e4bf01a7a.svg
paper_authors.fullname: Yuqiong Liu
paper_authors.type: user
paper_authors.user: lyq333
paper_authors._id: 683e7a6d97fd742a8edee1c5
paper_authors.name: An Yang
paper_authors._id: 683e7a6d97fd742a8edee1c6
paper_authors.name: Andrew Zhao
paper_authors._id: 683e7a6d97fd742a8edee1c7
paper_authors.name: Yang Yue
paper_authors._id: 683e7a6d97fd742a8edee1c8
paper_authors.name: Shiji Song
paper_authors._id: 683e7a6d97fd742a8edee1c9
paper_authors.name: Bowen Yu
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-06-03T04:30:38.648Z
paper_authors._id: 63d9d68c1cae35c27bf7a6a7
paper_authors.avatarUrl: /avatars/b5ad98cf269ae5f1fe90861fb4170fae.svg
paper_authors.fullname: Bowen Yu
paper_authors.type: user
paper_authors.user: Tigerph
paper_authors._id: 683e7a6d97fd742a8edee1ca
paper_authors.name: Gao Huang
paper_authors._id: 683e7a6d97fd742a8edee1cb
paper_authors.name: Junyang Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-06-03T09:39:54.278Z
paper_authors._id: 620760a26e3b7210c2ff1943
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/620760a26e3b7210c2ff1943/VC-rKqimF6yxGESNVlPoR.jpeg
paper_authors.fullname: Junyang Lin
paper_authors.type: user
paper_authors.user: JustinLin610
paper_title: Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective
    Reinforcement Learning for LLM Reasoning
paper_summary: Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a
  powerful approach to enhancing the reasoning capabilities of Large Language
  Models (LLMs), while its mechanisms are not yet well understood. In this work,
  we undertake a pioneering exploration of RLVR through the novel perspective of
  token entropy patterns, comprehensively analyzing how different tokens
  influence reasoning performance. By examining token entropy patterns in
  Chain-of-Thought (CoT) reasoning, we observe that only a small fraction of
  tokens exhibit high entropy, and these tokens act as critical forks that steer
  the model toward diverse reasoning pathways. Furthermore, studying how entropy
  patterns evolve during RLVR training reveals that RLVR largely adheres to the
  base model's entropy patterns, primarily adjusting the entropy of high-entropy
  tokens. These findings highlight the significance of high-entropy tokens (i.e.,
  forking tokens) to RLVR. We ultimately improve RLVR by restricting policy
  gradient updates to forking tokens and uncover a finding even beyond the 80/20
  rule: utilizing only 20% of the tokens while maintaining performance comparable
  to full-gradient updates on the Qwen3-8B base model and significantly
  surpassing full-gradient updates on the Qwen3-32B (+11.04 on AIME'25 and +7.71
  on AIME'24) and Qwen3-14B (+4.79 on AIME'25 and +5.21 on AIME'24) base models,
  highlighting a strong scaling trend. In contrast, training exclusively on the
  80% lowest-entropy tokens leads to a marked decline in performance. These
  findings indicate that the efficacy of RLVR primarily arises from optimizing
  the high-entropy tokens that decide reasoning directions. Collectively, our
  results highlight the potential to understand RLVR through a token-entropy
  perspective and optimize RLVR by leveraging high-entropy minority tokens to
  further improve LLM reasoning.
paper_ai_summary: Token entropy patterns are crucial in RLVR, with high-entropy tokens significantly impacting reasoning performance and model optimization.
paper_ai_keywords: Reinforcement Learning with Verifiable Rewards RLVR Large Language Models LLMs token entropy patterns Chain-of-Thought CoT reasoning high-entropy tokens policy gradient updates Qwen3-8B Qwen3-32B Qwen3-14B AIME