Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement
  Learning on the Base Model
We introduce Open-Reasoner-Zero, the first open source implementation of
large-scale reasoning-oriented RL training focusing on scalability, simplicity
and accessibility. Through extensive experiments, we demonstrate that a
minimalist approach, vanilla PPO with GAE (lambda=1, gamma=1) and
straightforward rule-based rewards, without any KL regularization, is
sufficient to scale up both response length and benchmark performance, similar
to the phenomenon observed in DeepSeek-R1-Zero. Using the same base model as
DeepSeek-R1-Zero-Qwen-32B, our implementation achieves superior performance on
AIME2024, MATH500, and the GPQA Diamond benchmark while demonstrating
remarkable efficiency -- requiring only a tenth of the training steps, compared
to DeepSeek-R1-Zero pipeline. In the spirit of open source, we release our
source code, parameter settings, training data, and model weights across
various sizes.
67eb762381e530baa56dc830
Jingcheng Hu
admin_assigned
2025-04-01T08:09:02.123Z
625026b7d2d191ac43320c5e
https://cdn-avatars.huggingface.co/v1/production/uploads/625026b7d2d191ac43320c5e/2ExzHlZ-Bk8SQMyBjeY6N.jpeg
Jingcheng Hu
user
reign12
67eb762381e530baa56dc831
Yinmin Zhang
admin_assigned
2025-04-01T08:09:09.884Z
664ae39ab5e5f95dc6209365
https://cdn-avatars.huggingface.co/v1/production/uploads/664ae39ab5e5f95dc6209365/8Z9ERYhX6URXh4si6jWGm.jpeg
Yinmin Zhang
user
YinminZhang
67eb762381e530baa56dc832
Qi Han
67eb762381e530baa56dc833
Daxin Jiang
admin_assigned
2025-04-01T08:09:22.914Z
60d4440fe648443279aaffd8
/avatars/bf7209c1f14ae120f5bfda5fda1301b7.svg
Daxin Jiang
user
djiang
67eb762381e530baa56dc834
Xiangyu Zhang
67eb762381e530baa56dc835
Heung-Yeung Shum
Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement
  Learning on the Base Model
We introduce Open-Reasoner-Zero, the first open source implementation of
large-scale reasoning-oriented RL training focusing on scalability, simplicity
and accessibility. Through extensive experiments, we demonstrate that a
minimalist approach, vanilla PPO with GAE (lambda=1, gamma=1) and
straightforward rule-based rewards, without any KL regularization, is
sufficient to scale up both response length and benchmark performance, similar
to the phenomenon observed in DeepSeek-R1-Zero. Using the same base model as
DeepSeek-R1-Zero-Qwen-32B, our implementation achieves superior performance on
AIME2024, MATH500, and the GPQA Diamond benchmark while demonstrating
remarkable efficiency -- requiring only a tenth of the training steps, compared
to DeepSeek-R1-Zero pipeline. In the spirit of open source, we release our
source code, parameter settings, training data, and model weights across
various sizes.
Open-Reasoner-Zero achieves superior performance on reasoning benchmarks using a minimalist PPO approach, requiring fewer training steps than DeepSeek-R1-Zero.
reinforcement learning (RL) PPO GAE benchmark performance AIME2024 MATH500 GPQA Diamond training steps DeepSeek-R1-Zero-Qwen-32B