Does Reinforcement Learning Really Incentivize Reasoning Capacity in
  LLMs Beyond the Base Model?
Reinforcement Learning with Verifiable Rewards (RLVR) has recently
demonstrated notable success in enhancing the reasoning capabilities of LLMs,
particularly in mathematics and programming tasks. It is widely believed that
RLVR enables LLMs to continuously self-improve, thus acquiring novel reasoning
abilities that exceed corresponding base models' capacity. In this study,
however, we critically re-examines this assumption by measuring the
pass@k metric with large values of k to explore the reasoning
capability boundary of the models across a wide range of model families and
benchmarks. Surprisingly, the RL does not, in fact, elicit fundamentally
new reasoning patterns. While RL-trained models outperform their base models at
smaller values of k (\eg, k=1), base models can achieve a comparable or
even higher pass@k score compared to their RL counterparts at large k
values. The reasoning paths generated by RL-trained models are already included
in the base models' sampling distribution, suggesting that most reasoning
abilities manifested in RL-trained models are already obtained by base models.
Further analysis shows that RL training boosts the performance by biasing the
model's output distribution toward paths that are more likely to yield rewards,
therefore sampling correct responses more efficiently. But this also results in
a narrower reasoning capability boundary compared to base models. Similar
results are observed in visual reasoning tasks trained with RLVR. Moreover, we
find that distillation can genuinely introduce new knowledge into the model,
different from RLVR. These findings underscore a critical limitation of RLVR in
advancing LLM reasoning abilities which requires us to fundamentally rethink
the impact of RL training in reasoning LLMs and the need of a better paradigm.
Project Page: https://limit-of-RLVR.github.io
6805b9ec7c5fa8020f595642
Yang Yue
claimed_verified
2025-04-22T09:59:45.493Z
649d475111592b1a765ac1a3
https://cdn-avatars.huggingface.co/v1/production/uploads/649d475111592b1a765ac1a3/rjORJjErJq-mthghan08U.jpeg
Yang Yue
user
Yang130
6805b9ec7c5fa8020f595643
Zhiqi Chen
claimed_verified
2025-04-22T09:52:56.179Z
67e82854d6032acbfa351d13
/avatars/b521257b74cc2c2816db06992fcd6770.svg
Zhiqi Chen
user
ZhiqiChen
6805b9ec7c5fa8020f595644
Rui Lu
claimed_verified
2025-04-22T09:59:47.731Z
64b77916964f6e7bf32bb428
/avatars/631b0324b4d713d351abbedb468658dc.svg
Rui Lu
user
RayLuTHU
6805b9ec7c5fa8020f595645
Andrew Zhao
claimed_verified
2025-04-22T09:52:53.907Z
630482fbce6b12280b18971d
/avatars/b07f31fd970d736bdf574d56da7a5634.svg
Andrew Zhao
user
andrewzh
6805b9ec7c5fa8020f595646
Zhaokai Wang
claimed_verified
2025-04-30T07:56:56.761Z
665d4b515fdfe8f923e347a7
/avatars/d114b24c02dadfca0a8aee104755a8ec.svg
Zhaokai Wang
user
wzk1015
6805b9ec7c5fa8020f595647
Yang Yue
6805b9ec7c5fa8020f595648
Shiji Song
6805b9ec7c5fa8020f595649
Gao Huang
Does Reinforcement Learning Really Incentivize Reasoning Capacity in
  LLMs Beyond the Base Model?
Reinforcement Learning with Verifiable Rewards (RLVR) has recently
demonstrated notable success in enhancing the reasoning capabilities of LLMs,
particularly in mathematics and programming tasks. It is widely believed that
RLVR enables LLMs to continuously self-improve, thus acquiring novel reasoning
abilities that exceed corresponding base models' capacity. In this study,
however, we critically re-examines this assumption by measuring the
pass@k metric with large values of k to explore the reasoning
capability boundary of the models across a wide range of model families and
benchmarks. Surprisingly, the RL does not, in fact, elicit fundamentally
new reasoning patterns. While RL-trained models outperform their base models at
smaller values of k (\eg, k=1), base models can achieve a comparable or
even higher pass@k score compared to their RL counterparts at large k
values. The reasoning paths generated by RL-trained models are already included
in the base models' sampling distribution, suggesting that most reasoning
abilities manifested in RL-trained models are already obtained by base models.
Further analysis shows that RL training boosts the performance by biasing the
model's output distribution toward paths that are more likely to yield rewards,
therefore sampling correct responses more efficiently. But this also results in
a narrower reasoning capability boundary compared to base models. Similar
results are observed in visual reasoning tasks trained with RLVR. Moreover, we
find that distillation can genuinely introduce new knowledge into the model,
different from RLVR. These findings underscore a critical limitation of RLVR in
advancing LLM reasoning abilities which requires us to fundamentally rethink
the impact of RL training in reasoning LLMs and the need of a better paradigm.
Project Page: https://limit-of-RLVR.github.io
Despite initial claims, reinforcement learning with verifiable rewards does not introduce fundamentally new reasoning abilities to LLMs, instead it enhances performance by biasing output distribution toward rewarded paths without expanding the reasoning boundary.
Reinforcement Learning with Verifiable Rewards RLVR pass\@k reasoning paths sampling distribution visual reasoning distillation