Stabilizing Knowledge, Promoting Reasoning: Dual-Token Constraints for
  RLVR
Reinforcement Learning with Verifiable Rewards (RLVR) has become an effective
post-training method for improving the reasoning abilities of Large Language
Models (LLMs), mainly by shaping higher-order behaviors such as reflection and
planning. However, previous RLVR algorithms often apply uniform training
signals to all tokens, without considering the different roles of low-entropy
knowledge-related tokens and high-entropy reasoning-related tokens. Some recent
methods try to separate these token types by gradient masking or asynchronous
updates, but these approaches may break semantic dependencies in the model
output and hinder effective learning. In this work, we propose Archer, an
entropy-aware RLVR approach with dual-token constraints and synchronous
updates. Specifically, our method applies weaker KL regularization and higher
clipping thresholds to reasoning tokens to encourage exploration, while using
stronger constraints on knowledge tokens to maintain factual knowledge.
Experimental results on several mathematical reasoning and code generation
benchmarks show that our approach significantly outperforms previous RLVR
methods, reaching or exceeding state-of-the-art performance among models of
comparable size. The code is available at
https://github.com/wizard-III/ArcherCodeR.
687efb4433947f780d9b4ae7
Jiakang Wang
687efb4433947f780d9b4ae8
Runze Liu
claimed_verified
2025-07-22T07:48:05.816Z
667187ba9ab144eb3ac43a1b
/avatars/db5558aa1c5160b9aee8b58573271959.svg
Runze Liu
user
RyanLiu112
687efb4433947f780d9b4ae9
Fuzheng Zhang
687efb4433947f780d9b4aea
Xiu Li
687efb4433947f780d9b4aeb
Guorui Zhou
Stabilizing Knowledge, Promoting Reasoning: Dual-Token Constraints for
  RLVR
Reinforcement Learning with Verifiable Rewards (RLVR) has become an effective
post-training method for improving the reasoning abilities of Large Language
Models (LLMs), mainly by shaping higher-order behaviors such as reflection and
planning. However, previous RLVR algorithms often apply uniform training
signals to all tokens, without considering the different roles of low-entropy
knowledge-related tokens and high-entropy reasoning-related tokens. Some recent
methods try to separate these token types by gradient masking or asynchronous
updates, but these approaches may break semantic dependencies in the model
output and hinder effective learning. In this work, we propose Archer, an
entropy-aware RLVR approach with dual-token constraints and synchronous
updates. Specifically, our method applies weaker KL regularization and higher
clipping thresholds to reasoning tokens to encourage exploration, while using
stronger constraints on knowledge tokens to maintain factual knowledge.
Experimental results on several mathematical reasoning and code generation
benchmarks show that our approach significantly outperforms previous RLVR
methods, reaching or exceeding state-of-the-art performance among models of
comparable size. The code is available at
https://github.com/wizard-III/ArcherCodeR.
Archer, an entropy-aware RLVR approach with dual-token constraints and synchronous updates, enhances LLM reasoning abilities by differentiating between knowledge and reasoning tokens, achieving state-of-the-art performance on mathematical reasoning and code generation benchmarks.
Reinforcement Learning with Verifiable Rewards (RLVR) Large Language Models (LLMs) entropy-aware RLVR dual-token constraints synchronous updates KL regularization clipping thresholds mathematical reasoning code generation