Understanding R1-Zero-Like Training: A Critical Perspective
DeepSeek-R1-Zero has shown that reinforcement learning (RL) at scale can
directly enhance the reasoning capabilities of LLMs without supervised
fine-tuning. In this work, we critically examine R1-Zero-like training by
analyzing its two core components: base models and RL. We investigate a wide
range of base models, including DeepSeek-V3-Base, to understand how pretraining
characteristics influence RL performance. Our analysis reveals that
DeepSeek-V3-Base already exhibit ''Aha moment'', while Qwen2.5 base models
demonstrate strong reasoning capabilities even without prompt templates,
suggesting potential pretraining biases. Additionally, we identify an
optimization bias in Group Relative Policy Optimization (GRPO), which
artificially increases response length (especially for incorrect outputs)
during training. To address this, we introduce Dr. GRPO, an unbiased
optimization method that improves token efficiency while maintaining reasoning
performance. Leveraging these insights, we present a minimalist R1-Zero recipe
that achieves 43.3% accuracy on AIME 2024 with a 7B base model, establishing a
new state-of-the-art. Our code is available at
https://github.com/sail-sg/understand-r1-zero.
67e97f581cb6fc648f642a05
Zichen Liu
claimed_verified
2025-04-03T08:29:10.035Z
65f5392c68b8e0cb3c9977a2
/avatars/aa64772475098e8a135c13072fde6744.svg
Zichen
user
lkevinzc
67e97f581cb6fc648f642a06
Changyu Chen
claimed_verified
2025-04-03T08:29:07.988Z
64e416dc54e18f390ef79ba4
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/5n01J00ZaVRrebsON8iYA.jpeg
Changyu Chen
user
Cameron-Chen
67e97f581cb6fc648f642a07
Wenjun Li
claimed_verified
2025-06-11T08:37:39.278Z
6622245224f3842a31f7c58a
https://cdn-avatars.huggingface.co/v1/production/uploads/6622245224f3842a31f7c58a/EcvwtHamYwMdzRT9pevdE.jpeg
wenjun
user
wenjun-li
67e97f581cb6fc648f642a08
Penghui Qi
claimed_verified
2025-04-03T08:29:05.240Z
63885f1d0bebb233d8ad6e5b
https://cdn-avatars.huggingface.co/v1/production/uploads/1669881620925-noauth.jpeg
Penghui Qi
user
QPHutu
67e97f581cb6fc648f642a09
Tianyu Pang
admin_assigned
2025-04-03T13:50:22.635Z
63d91b6d255ef6add20e1b38
https://cdn-avatars.huggingface.co/v1/production/uploads/1675921369867-63d91b6d255ef6add20e1b38.jpeg
Tianyu Pang
user
P2333
67e97f581cb6fc648f642a0a
Chao Du
admin_assigned
2025-04-03T13:50:46.967Z
632407c892e07e3ca20aca28
/avatars/23b51b37b12b51a0947f687d1de4d3b5.svg
Chao Du
user
duchao
67e97f581cb6fc648f642a0b
Wee Sun Lee
67e97f581cb6fc648f642a0c
Min Lin
Understanding R1-Zero-Like Training: A Critical Perspective
DeepSeek-R1-Zero has shown that reinforcement learning (RL) at scale can
directly enhance the reasoning capabilities of LLMs without supervised
fine-tuning. In this work, we critically examine R1-Zero-like training by
analyzing its two core components: base models and RL. We investigate a wide
range of base models, including DeepSeek-V3-Base, to understand how pretraining
characteristics influence RL performance. Our analysis reveals that
DeepSeek-V3-Base already exhibit ''Aha moment'', while Qwen2.5 base models
demonstrate strong reasoning capabilities even without prompt templates,
suggesting potential pretraining biases. Additionally, we identify an
optimization bias in Group Relative Policy Optimization (GRPO), which
artificially increases response length (especially for incorrect outputs)
during training. To address this, we introduce Dr. GRPO, an unbiased
optimization method that improves token efficiency while maintaining reasoning
performance. Leveraging these insights, we present a minimalist R1-Zero recipe
that achieves 43.3% accuracy on AIME 2024 with a 7B base model, establishing a
new state-of-the-art. Our code is available at
https://github.com/sail-sg/understand-r1-zero.
A study of R1-Zero training identifies pretraining effects on RL performance and proffers Dr. GRPO to enhance token efficiency, achieving superior accuracy on AIME 2024.
reinforcement learning RL LLMs DeepSeek-V3-Base Qwen2.5 Group Relative Policy Optimization GRPO Dr. GRPO token efficiency AIME 2024