ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in
  Large Language Models
Recent advances in reasoning-centric language models have highlighted
reinforcement learning (RL) as a promising method for aligning models with
verifiable rewards. However, it remains contentious whether RL truly expands a
model's reasoning capabilities or merely amplifies high-reward outputs already
latent in the base model's distribution, and whether continually scaling up RL
compute reliably leads to improved reasoning performance. In this work, we
challenge prevailing assumptions by demonstrating that prolonged RL (ProRL)
training can uncover novel reasoning strategies that are inaccessible to base
models, even under extensive sampling. We introduce ProRL, a novel training
methodology that incorporates KL divergence control, reference policy
resetting, and a diverse suite of tasks. Our empirical analysis reveals that
RL-trained models consistently outperform base models across a wide range of
pass@k evaluations, including scenarios where base models fail entirely
regardless of the number of attempts. We further show that reasoning boundary
improvements correlates strongly with task competence of base model and
training duration, suggesting that RL can explore and populate new regions of
solution space over time. These findings offer new insights into the conditions
under which RL meaningfully expands reasoning boundaries in language models and
establish a foundation for future work on long-horizon RL for reasoning. We
release model weights to support further research:
https://huggingface.co/nvidia/Nemotron-Research-Reasoning-Qwen-1.5B
683d2d05ae87a04bca311b22
Mingjie Liu
683d2d05ae87a04bca311b23
Shizhe Diao
683d2d05ae87a04bca311b24
Ximing Lu
683d2d05ae87a04bca311b25
Jian Hu
claimed_verified
2025-06-05T07:28:40.779Z
63f6c04ac96958470d1e9043
/avatars/da46cdd9e21498e120ca91b67bfbfb5e.svg
Jian Hu
user
chuyi777
683d2d05ae87a04bca311b26
Xin Dong
683d2d05ae87a04bca311b27
Yejin Choi
683d2d05ae87a04bca311b28
Jan Kautz
683d2d05ae87a04bca311b29
Yi Dong
ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in
  Large Language Models
Recent advances in reasoning-centric language models have highlighted
reinforcement learning (RL) as a promising method for aligning models with
verifiable rewards. However, it remains contentious whether RL truly expands a
model's reasoning capabilities or merely amplifies high-reward outputs already
latent in the base model's distribution, and whether continually scaling up RL
compute reliably leads to improved reasoning performance. In this work, we
challenge prevailing assumptions by demonstrating that prolonged RL (ProRL)
training can uncover novel reasoning strategies that are inaccessible to base
models, even under extensive sampling. We introduce ProRL, a novel training
methodology that incorporates KL divergence control, reference policy
resetting, and a diverse suite of tasks. Our empirical analysis reveals that
RL-trained models consistently outperform base models across a wide range of
pass@k evaluations, including scenarios where base models fail entirely
regardless of the number of attempts. We further show that reasoning boundary
improvements correlates strongly with task competence of base model and
training duration, suggesting that RL can explore and populate new regions of
solution space over time. These findings offer new insights into the conditions
under which RL meaningfully expands reasoning boundaries in language models and
establish a foundation for future work on long-horizon RL for reasoning. We
release model weights to support further research:
https://huggingface.co/nvidia/Nemotron-Research-Reasoning-Qwen-1.5B
Prolonged reinforcement learning training (ProRL) uncovers novel reasoning strategies in language models, outperforming base models and suggesting meaningful expansion of reasoning capabilities.
reinforcement learning RL ProRL KL divergence control reference policy resetting pass@k evaluations reasoning boundary improvements task competence long-horizon RL