title: Does Reinforcement Learning Really Incentivize Reasoning Capacity in
    LLMs Beyond the Base Model?
summary: Reinforcement Learning with Verifiable Rewards (RLVR) has recently
  demonstrated notable success in enhancing the reasoning capabilities of LLMs,
  particularly in mathematics and programming tasks. It is widely believed that
  RLVR enables LLMs to continuously self-improve, thus acquiring novel reasoning
  abilities that exceed corresponding base models' capacity. In this study,
  however, we critically re-examines this assumption by measuring the
  pass@k metric with large values of k to explore the reasoning
  capability boundary of the models across a wide range of model families and
  benchmarks. Surprisingly, the RL does not, in fact, elicit fundamentally
  new reasoning patterns. While RL-trained models outperform their base models at
  smaller values of k (\eg, k=1), base models can achieve a comparable or
  even higher pass@k score compared to their RL counterparts at large k
  values. The reasoning paths generated by RL-trained models are already included
  in the base models' sampling distribution, suggesting that most reasoning
  abilities manifested in RL-trained models are already obtained by base models.
  Further analysis shows that RL training boosts the performance by biasing the
  model's output distribution toward paths that are more likely to yield rewards,
  therefore sampling correct responses more efficiently. But this also results in
  a narrower reasoning capability boundary compared to base models. Similar
  results are observed in visual reasoning tasks trained with RLVR. Moreover, we
  find that distillation can genuinely introduce new knowledge into the model,
  different from RLVR. These findings underscore a critical limitation of RLVR in
  advancing LLM reasoning abilities which requires us to fundamentally rethink
  the impact of RL training in reasoning LLMs and the need of a better paradigm.
  Project Page: https://limit-of-RLVR.github.io
paper_authors._id: 6805b9ec7c5fa8020f595642
paper_authors.name: Yang Yue
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-22T09:59:45.493Z
paper_authors._id: 649d475111592b1a765ac1a3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/649d475111592b1a765ac1a3/rjORJjErJq-mthghan08U.jpeg
paper_authors.fullname: Yang Yue
paper_authors.type: user
paper_authors.user: Yang130
paper_authors._id: 6805b9ec7c5fa8020f595643
paper_authors.name: Zhiqi Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-22T09:52:56.179Z
paper_authors._id: 67e82854d6032acbfa351d13
paper_authors.avatarUrl: /avatars/b521257b74cc2c2816db06992fcd6770.svg
paper_authors.fullname: Zhiqi Chen
paper_authors.type: user
paper_authors.user: ZhiqiChen
paper_authors._id: 6805b9ec7c5fa8020f595644
paper_authors.name: Rui Lu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-22T09:59:47.731Z
paper_authors._id: 64b77916964f6e7bf32bb428
paper_authors.avatarUrl: /avatars/631b0324b4d713d351abbedb468658dc.svg
paper_authors.fullname: Rui Lu
paper_authors.type: user
paper_authors.user: RayLuTHU
paper_authors._id: 6805b9ec7c5fa8020f595645
paper_authors.name: Andrew Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-22T09:52:53.907Z
paper_authors._id: 630482fbce6b12280b18971d
paper_authors.avatarUrl: /avatars/b07f31fd970d736bdf574d56da7a5634.svg
paper_authors.fullname: Andrew Zhao
paper_authors.type: user
paper_authors.user: andrewzh
paper_authors._id: 6805b9ec7c5fa8020f595646
paper_authors.name: Zhaokai Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-30T07:56:56.761Z
paper_authors._id: 665d4b515fdfe8f923e347a7
paper_authors.avatarUrl: /avatars/d114b24c02dadfca0a8aee104755a8ec.svg
paper_authors.fullname: Zhaokai Wang
paper_authors.type: user
paper_authors.user: wzk1015
paper_authors._id: 6805b9ec7c5fa8020f595647
paper_authors.name: Yang Yue
paper_authors._id: 6805b9ec7c5fa8020f595648
paper_authors.name: Shiji Song
paper_authors._id: 6805b9ec7c5fa8020f595649
paper_authors.name: Gao Huang
paper_title: Does Reinforcement Learning Really Incentivize Reasoning Capacity in
    LLMs Beyond the Base Model?
paper_summary: Reinforcement Learning with Verifiable Rewards (RLVR) has recently
  demonstrated notable success in enhancing the reasoning capabilities of LLMs,
  particularly in mathematics and programming tasks. It is widely believed that
  RLVR enables LLMs to continuously self-improve, thus acquiring novel reasoning
  abilities that exceed corresponding base models' capacity. In this study,
  however, we critically re-examines this assumption by measuring the
  pass@k metric with large values of k to explore the reasoning
  capability boundary of the models across a wide range of model families and
  benchmarks. Surprisingly, the RL does not, in fact, elicit fundamentally
  new reasoning patterns. While RL-trained models outperform their base models at
  smaller values of k (\eg, k=1), base models can achieve a comparable or
  even higher pass@k score compared to their RL counterparts at large k
  values. The reasoning paths generated by RL-trained models are already included
  in the base models' sampling distribution, suggesting that most reasoning
  abilities manifested in RL-trained models are already obtained by base models.
  Further analysis shows that RL training boosts the performance by biasing the
  model's output distribution toward paths that are more likely to yield rewards,
  therefore sampling correct responses more efficiently. But this also results in
  a narrower reasoning capability boundary compared to base models. Similar
  results are observed in visual reasoning tasks trained with RLVR. Moreover, we
  find that distillation can genuinely introduce new knowledge into the model,
  different from RLVR. These findings underscore a critical limitation of RLVR in
  advancing LLM reasoning abilities which requires us to fundamentally rethink
  the impact of RL training in reasoning LLMs and the need of a better paradigm.
  Project Page: https://limit-of-RLVR.github.io
paper_ai_summary: Despite initial claims, reinforcement learning with verifiable rewards does not introduce fundamentally new reasoning abilities to LLMs, instead it enhances performance by biasing output distribution toward rewarded paths without expanding the reasoning boundary.
paper_ai_keywords: Reinforcement Learning with Verifiable Rewards RLVR pass\@k reasoning paths sampling distribution visual reasoning distillation