title: Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement
    Learning on the Base Model
summary: We introduce Open-Reasoner-Zero, the first open source implementation of
  large-scale reasoning-oriented RL training focusing on scalability, simplicity
  and accessibility. Through extensive experiments, we demonstrate that a
  minimalist approach, vanilla PPO with GAE (lambda=1, gamma=1) and
  straightforward rule-based rewards, without any KL regularization, is
  sufficient to scale up both response length and benchmark performance, similar
  to the phenomenon observed in DeepSeek-R1-Zero. Using the same base model as
  DeepSeek-R1-Zero-Qwen-32B, our implementation achieves superior performance on
  AIME2024, MATH500, and the GPQA Diamond benchmark while demonstrating
  remarkable efficiency -- requiring only a tenth of the training steps, compared
  to DeepSeek-R1-Zero pipeline. In the spirit of open source, we release our
  source code, parameter settings, training data, and model weights across
  various sizes.
paper_authors._id: 67eb762381e530baa56dc830
paper_authors.name: Jingcheng Hu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-01T08:09:02.123Z
paper_authors._id: 625026b7d2d191ac43320c5e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/625026b7d2d191ac43320c5e/2ExzHlZ-Bk8SQMyBjeY6N.jpeg
paper_authors.fullname: Jingcheng Hu
paper_authors.type: user
paper_authors.user: reign12
paper_authors._id: 67eb762381e530baa56dc831
paper_authors.name: Yinmin Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-01T08:09:09.884Z
paper_authors._id: 664ae39ab5e5f95dc6209365
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/664ae39ab5e5f95dc6209365/8Z9ERYhX6URXh4si6jWGm.jpeg
paper_authors.fullname: Yinmin Zhang
paper_authors.type: user
paper_authors.user: YinminZhang
paper_authors._id: 67eb762381e530baa56dc832
paper_authors.name: Qi Han
paper_authors._id: 67eb762381e530baa56dc833
paper_authors.name: Daxin Jiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-01T08:09:22.914Z
paper_authors._id: 60d4440fe648443279aaffd8
paper_authors.avatarUrl: /avatars/bf7209c1f14ae120f5bfda5fda1301b7.svg
paper_authors.fullname: Daxin Jiang
paper_authors.type: user
paper_authors.user: djiang
paper_authors._id: 67eb762381e530baa56dc834
paper_authors.name: Xiangyu Zhang
paper_authors._id: 67eb762381e530baa56dc835
paper_authors.name: Heung-Yeung Shum
paper_title: Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement
    Learning on the Base Model
paper_summary: We introduce Open-Reasoner-Zero, the first open source implementation of
  large-scale reasoning-oriented RL training focusing on scalability, simplicity
  and accessibility. Through extensive experiments, we demonstrate that a
  minimalist approach, vanilla PPO with GAE (lambda=1, gamma=1) and
  straightforward rule-based rewards, without any KL regularization, is
  sufficient to scale up both response length and benchmark performance, similar
  to the phenomenon observed in DeepSeek-R1-Zero. Using the same base model as
  DeepSeek-R1-Zero-Qwen-32B, our implementation achieves superior performance on
  AIME2024, MATH500, and the GPQA Diamond benchmark while demonstrating
  remarkable efficiency -- requiring only a tenth of the training steps, compared
  to DeepSeek-R1-Zero pipeline. In the spirit of open source, we release our
  source code, parameter settings, training data, and model weights across
  various sizes.
paper_ai_summary: Open-Reasoner-Zero achieves superior performance on reasoning benchmarks using a minimalist PPO approach, requiring fewer training steps than DeepSeek-R1-Zero.
paper_ai_keywords: reinforcement learning (RL) PPO GAE benchmark performance AIME2024 MATH500 GPQA Diamond training steps DeepSeek-R1-Zero-Qwen-32B