Reasoning with Exploration: An Entropy Perspective
Balancing exploration and exploitation is a central goal in reinforcement
learning (RL). Despite recent advances in enhancing language model (LM)
reasoning, most methods lean toward exploitation, and increasingly encounter
performance plateaus. In this work, we revisit entropy -- a signal of
exploration in RL -- and examine its relationship to exploratory reasoning in
LMs. Through empirical analysis, we uncover strong positive correlations
between high-entropy regions and three types of exploratory reasoning actions:
(1) pivotal tokens that determine or connect logical steps, (2) reflective
actions such as self-verification and correction, and (3) rare behaviors
under-explored by the base LMs. Motivated by this, we introduce a minimal
modification to standard RL with only one line of code: augmenting the
advantage function with an entropy-based term. Unlike traditional
maximum-entropy methods which encourage exploration by promoting uncertainty,
we encourage exploration by promoting longer and deeper reasoning chains.
Notably, our method achieves significant gains on the Pass@K metric -- an
upper-bound estimator of LM reasoning capabilities -- even when evaluated with
extremely large K values, pushing the boundaries of LM reasoning.
685239610164cd1316710553
Daixuan Cheng
extracted_confirmed
2025-06-18T06:57:21.864Z
649e6761f9134a06ed1e0cea
https://cdn-avatars.huggingface.co/v1/production/uploads/649e6761f9134a06ed1e0cea/XNeKceE8xSwI0xWwWUwwJ.jpeg
Daixuan Cheng
user
daixuancheng
685239610164cd1316710554
Shaohan Huang
685239610164cd1316710555
Xuekai Zhu
685239610164cd1316710556
Bo Dai
685239610164cd1316710557
Wayne Xin Zhao
685239610164cd1316710558
Zhenliang Zhang
685239610164cd1316710559
Furu Wei
Reasoning with Exploration: An Entropy Perspective
Balancing exploration and exploitation is a central goal in reinforcement
learning (RL). Despite recent advances in enhancing language model (LM)
reasoning, most methods lean toward exploitation, and increasingly encounter
performance plateaus. In this work, we revisit entropy -- a signal of
exploration in RL -- and examine its relationship to exploratory reasoning in
LMs. Through empirical analysis, we uncover strong positive correlations
between high-entropy regions and three types of exploratory reasoning actions:
(1) pivotal tokens that determine or connect logical steps, (2) reflective
actions such as self-verification and correction, and (3) rare behaviors
under-explored by the base LMs. Motivated by this, we introduce a minimal
modification to standard RL with only one line of code: augmenting the
advantage function with an entropy-based term. Unlike traditional
maximum-entropy methods which encourage exploration by promoting uncertainty,
we encourage exploration by promoting longer and deeper reasoning chains.
Notably, our method achieves significant gains on the Pass@K metric -- an
upper-bound estimator of LM reasoning capabilities -- even when evaluated with
extremely large K values, pushing the boundaries of LM reasoning.
Introducing an entropy-based term to the advantage function in reinforcement learning enhances exploratory reasoning in language models, leading to improved performance on complex reasoning tasks.
reinforcement learning entropy exploratory reasoning pivotal tokens reflective actions rare behaviors advantage function Pass@K