title: ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in
    Large Language Models
summary: Recent advances in reasoning-centric language models have highlighted
  reinforcement learning (RL) as a promising method for aligning models with
  verifiable rewards. However, it remains contentious whether RL truly expands a
  model's reasoning capabilities or merely amplifies high-reward outputs already
  latent in the base model's distribution, and whether continually scaling up RL
  compute reliably leads to improved reasoning performance. In this work, we
  challenge prevailing assumptions by demonstrating that prolonged RL (ProRL)
  training can uncover novel reasoning strategies that are inaccessible to base
  models, even under extensive sampling. We introduce ProRL, a novel training
  methodology that incorporates KL divergence control, reference policy
  resetting, and a diverse suite of tasks. Our empirical analysis reveals that
  RL-trained models consistently outperform base models across a wide range of
  pass@k evaluations, including scenarios where base models fail entirely
  regardless of the number of attempts. We further show that reasoning boundary
  improvements correlates strongly with task competence of base model and
  training duration, suggesting that RL can explore and populate new regions of
  solution space over time. These findings offer new insights into the conditions
  under which RL meaningfully expands reasoning boundaries in language models and
  establish a foundation for future work on long-horizon RL for reasoning. We
  release model weights to support further research:
  https://huggingface.co/nvidia/Nemotron-Research-Reasoning-Qwen-1.5B
paper_authors._id: 683d2d05ae87a04bca311b22
paper_authors.name: Mingjie Liu
paper_authors._id: 683d2d05ae87a04bca311b23
paper_authors.name: Shizhe Diao
paper_authors._id: 683d2d05ae87a04bca311b24
paper_authors.name: Ximing Lu
paper_authors._id: 683d2d05ae87a04bca311b25
paper_authors.name: Jian Hu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-05T07:28:40.779Z
paper_authors._id: 63f6c04ac96958470d1e9043
paper_authors.avatarUrl: /avatars/da46cdd9e21498e120ca91b67bfbfb5e.svg
paper_authors.fullname: Jian Hu
paper_authors.type: user
paper_authors.user: chuyi777
paper_authors._id: 683d2d05ae87a04bca311b26
paper_authors.name: Xin Dong
paper_authors._id: 683d2d05ae87a04bca311b27
paper_authors.name: Yejin Choi
paper_authors._id: 683d2d05ae87a04bca311b28
paper_authors.name: Jan Kautz
paper_authors._id: 683d2d05ae87a04bca311b29
paper_authors.name: Yi Dong
paper_title: ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in
    Large Language Models
paper_summary: Recent advances in reasoning-centric language models have highlighted
  reinforcement learning (RL) as a promising method for aligning models with
  verifiable rewards. However, it remains contentious whether RL truly expands a
  model's reasoning capabilities or merely amplifies high-reward outputs already
  latent in the base model's distribution, and whether continually scaling up RL
  compute reliably leads to improved reasoning performance. In this work, we
  challenge prevailing assumptions by demonstrating that prolonged RL (ProRL)
  training can uncover novel reasoning strategies that are inaccessible to base
  models, even under extensive sampling. We introduce ProRL, a novel training
  methodology that incorporates KL divergence control, reference policy
  resetting, and a diverse suite of tasks. Our empirical analysis reveals that
  RL-trained models consistently outperform base models across a wide range of
  pass@k evaluations, including scenarios where base models fail entirely
  regardless of the number of attempts. We further show that reasoning boundary
  improvements correlates strongly with task competence of base model and
  training duration, suggesting that RL can explore and populate new regions of
  solution space over time. These findings offer new insights into the conditions
  under which RL meaningfully expands reasoning boundaries in language models and
  establish a foundation for future work on long-horizon RL for reasoning. We
  release model weights to support further research:
  https://huggingface.co/nvidia/Nemotron-Research-Reasoning-Qwen-1.5B
paper_ai_summary: Prolonged reinforcement learning training (ProRL) uncovers novel reasoning strategies in language models, outperforming base models and suggesting meaningful expansion of reasoning capabilities.
paper_ai_keywords: reinforcement learning RL ProRL KL divergence control reference policy resetting pass@k evaluations reasoning boundary improvements task competence long-horizon RL