Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes
  Correct Reasoning in Base LLMs
Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a
promising paradigm for advancing the reasoning capabilities of Large Language
Models (LLMs). However, a critical paradox clouds its efficacy: RLVR-tuned
models often underperform their base models on the Pass@K metric for
solution-finding, leading to the hypothesis that RLVR merely re-weights
existing reasoning paths at the cost of reasoning diversity. In this work, we
resolve this contradiction by identifying the source of the problem: the
Pass@K metric itself is a flawed measure of reasoning, as it credits correct
final answers that probably arise from inaccurate or incomplete chains of
thought (CoTs). To address this, we introduce a more precise evaluation metric,
CoT-Pass@K, which mandates that both the reasoning path and the final
answer be correct. We provide a new theoretical foundation that formalizes how
RLVR, unlike traditional RL, is uniquely structured to incentivize logical
integrity. Our empirical results are supportive: using CoT-Pass@K, we
observe that RLVR can incentivize the generalization of correct reasoning for
all values of K. Furthermore, by analyzing the training dynamics, we find
that this enhanced reasoning capability emerges early in the training process
and smoothly generalizes. Our work provides a clear perspective on the role of
RLVR, offers a more reliable method for its evaluation, and confirms its
potential to genuinely advance machine reasoning.
68521c2a0164cd131671046b
Xumeng Wen
claimed_verified
2025-06-18T12:15:48.128Z
669f940b3b09946711e20c52
/avatars/27044caec57d8d68d700208fae78b6c6.svg
XumengWen
user
XumengWen
68521c2a0164cd131671046c
Zihan Liu
68521c2a0164cd131671046d
Shun Zheng
extracted_confirmed
2025-06-18T02:37:06.379Z
64a7a2bad001860e0c34f7f2
/avatars/2433104071e4ae1c3e2d755d81d7964b.svg
Shun Zheng
user
shun-zheng
68521c2a0164cd131671046e
Zhijian Xu
claimed_verified
2025-06-18T12:15:46.044Z
67d7c0e0cb3e80af0d13660a
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/hCmEo__IWO_R_Ps8Q52Os.png
zhijianxu
user
VEWOXIC
68521c2a0164cd131671046f
Shengyu Ye
68521c2a0164cd1316710470
Zhirong Wu
68521c2a0164cd1316710471
Xiao Liang
claimed_verified
2025-06-18T12:16:18.590Z
6560763e152b659e623865ae
https://cdn-avatars.huggingface.co/v1/production/uploads/6560763e152b659e623865ae/cTT2jGnPU_8XMrUTvqZ2h.jpeg
Xiao Liang
user
MasterVito
68521c2a0164cd1316710472
Yang Wang
claimed_verified
2025-06-18T12:16:20.452Z
604714a0c82d59b7347b55ae
https://cdn-avatars.huggingface.co/v1/production/uploads/604714a0c82d59b7347b55ae/WZFKDDUPi8JS0BK8t7mIv.jpeg
YangWang92
user
yangwang92
68521c2a0164cd1316710473
Junjie Li
68521c2a0164cd1316710474
Ziming Miao
68521c2a0164cd1316710475
Jiang Bian
claimed_verified
2025-06-19T09:10:31.848Z
683843c8a9bdf98f92657042
/avatars/4983876f615fa830cfc8f2cb5a00fa08.svg
Jiang Bian
user
jiangbian22
68521c2a0164cd1316710476
Mao Yang
Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes
  Correct Reasoning in Base LLMs
Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a
promising paradigm for advancing the reasoning capabilities of Large Language
Models (LLMs). However, a critical paradox clouds its efficacy: RLVR-tuned
models often underperform their base models on the Pass@K metric for
solution-finding, leading to the hypothesis that RLVR merely re-weights
existing reasoning paths at the cost of reasoning diversity. In this work, we
resolve this contradiction by identifying the source of the problem: the
Pass@K metric itself is a flawed measure of reasoning, as it credits correct
final answers that probably arise from inaccurate or incomplete chains of
thought (CoTs). To address this, we introduce a more precise evaluation metric,
CoT-Pass@K, which mandates that both the reasoning path and the final
answer be correct. We provide a new theoretical foundation that formalizes how
RLVR, unlike traditional RL, is uniquely structured to incentivize logical
integrity. Our empirical results are supportive: using CoT-Pass@K, we
observe that RLVR can incentivize the generalization of correct reasoning for
all values of K. Furthermore, by analyzing the training dynamics, we find
that this enhanced reasoning capability emerges early in the training process
and smoothly generalizes. Our work provides a clear perspective on the role of
RLVR, offers a more reliable method for its evaluation, and confirms its
potential to genuinely advance machine reasoning.
RLVR advances machine reasoning by incentivizing correct and logical thought chains, addressing limitations identified by a more precise evaluation metric, $CoT$-$Pass@K$.
Reinforcement Learning with Verifiable Rewards LLMS Pass@K chains of thought CoT-Pass@K logical integrity machine reasoning training dynamics