title: Understanding R1-Zero-Like Training: A Critical Perspective
summary: DeepSeek-R1-Zero has shown that reinforcement learning (RL) at scale can
  directly enhance the reasoning capabilities of LLMs without supervised
  fine-tuning. In this work, we critically examine R1-Zero-like training by
  analyzing its two core components: base models and RL. We investigate a wide
  range of base models, including DeepSeek-V3-Base, to understand how pretraining
  characteristics influence RL performance. Our analysis reveals that
  DeepSeek-V3-Base already exhibit ''Aha moment'', while Qwen2.5 base models
  demonstrate strong reasoning capabilities even without prompt templates,
  suggesting potential pretraining biases. Additionally, we identify an
  optimization bias in Group Relative Policy Optimization (GRPO), which
  artificially increases response length (especially for incorrect outputs)
  during training. To address this, we introduce Dr. GRPO, an unbiased
  optimization method that improves token efficiency while maintaining reasoning
  performance. Leveraging these insights, we present a minimalist R1-Zero recipe
  that achieves 43.3% accuracy on AIME 2024 with a 7B base model, establishing a
  new state-of-the-art. Our code is available at
  https://github.com/sail-sg/understand-r1-zero.
paper_authors._id: 67e97f581cb6fc648f642a05
paper_authors.name: Zichen Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-03T08:29:10.035Z
paper_authors._id: 65f5392c68b8e0cb3c9977a2
paper_authors.avatarUrl: /avatars/aa64772475098e8a135c13072fde6744.svg
paper_authors.fullname: Zichen
paper_authors.type: user
paper_authors.user: lkevinzc
paper_authors._id: 67e97f581cb6fc648f642a06
paper_authors.name: Changyu Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-03T08:29:07.988Z
paper_authors._id: 64e416dc54e18f390ef79ba4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/5n01J00ZaVRrebsON8iYA.jpeg
paper_authors.fullname: Changyu Chen
paper_authors.type: user
paper_authors.user: Cameron-Chen
paper_authors._id: 67e97f581cb6fc648f642a07
paper_authors.name: Wenjun Li
paper_authors._id: 67e97f581cb6fc648f642a08
paper_authors.name: Penghui Qi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-03T08:29:05.240Z
paper_authors._id: 63885f1d0bebb233d8ad6e5b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1669881620925-noauth.jpeg
paper_authors.fullname: Penghui Qi
paper_authors.type: user
paper_authors.user: QPHutu
paper_authors._id: 67e97f581cb6fc648f642a09
paper_authors.name: Tianyu Pang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-03T13:50:22.635Z
paper_authors._id: 63d91b6d255ef6add20e1b38
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1675921369867-63d91b6d255ef6add20e1b38.jpeg
paper_authors.fullname: Tianyu Pang
paper_authors.type: user
paper_authors.user: P2333
paper_authors._id: 67e97f581cb6fc648f642a0a
paper_authors.name: Chao Du
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-03T13:50:46.967Z
paper_authors._id: 632407c892e07e3ca20aca28
paper_authors.avatarUrl: /avatars/23b51b37b12b51a0947f687d1de4d3b5.svg
paper_authors.fullname: Chao Du
paper_authors.type: user
paper_authors.user: duchao
paper_authors._id: 67e97f581cb6fc648f642a0b
paper_authors.name: Wee Sun Lee
paper_authors._id: 67e97f581cb6fc648f642a0c
paper_authors.name: Min Lin
paper_title: Understanding R1-Zero-Like Training: A Critical Perspective
paper_summary: DeepSeek-R1-Zero has shown that reinforcement learning (RL) at scale can
  directly enhance the reasoning capabilities of LLMs without supervised
  fine-tuning. In this work, we critically examine R1-Zero-like training by
  analyzing its two core components: base models and RL. We investigate a wide
  range of base models, including DeepSeek-V3-Base, to understand how pretraining
  characteristics influence RL performance. Our analysis reveals that
  DeepSeek-V3-Base already exhibit ''Aha moment'', while Qwen2.5 base models
  demonstrate strong reasoning capabilities even without prompt templates,
  suggesting potential pretraining biases. Additionally, we identify an
  optimization bias in Group Relative Policy Optimization (GRPO), which
  artificially increases response length (especially for incorrect outputs)
  during training. To address this, we introduce Dr. GRPO, an unbiased
  optimization method that improves token efficiency while maintaining reasoning
  performance. Leveraging these insights, we present a minimalist R1-Zero recipe
  that achieves 43.3% accuracy on AIME 2024 with a 7B base model, establishing a
  new state-of-the-art. Our code is available at
  https://github.com/sail-sg/understand-r1-zero.
paper_ai_summary: A study of R1-Zero training identifies pretraining effects on RL performance and proffers Dr. GRPO to enhance token efficiency, achieving superior accuracy on AIME 2024.
paper_ai_keywords: reinforcement learning RL LLMs DeepSeek-V3-Base Qwen2.5 Group Relative Policy Optimization GRPO Dr. GRPO token efficiency AIME 2024