title: SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for
    Open Base Models in the Wild
summary: DeepSeek-R1 has shown that long chain-of-thought (CoT) reasoning can
  naturally emerge through a simple reinforcement learning (RL) framework with
  rule-based rewards, where the training may directly start from the base
  models-a paradigm referred to as zero RL training. Most recent efforts to
  reproduce zero RL training have primarily focused on the Qwen2.5 model series,
  which may not be representative as we find the base models already exhibit
  strong instruction-following and self-reflection abilities. In this work, we
  investigate zero RL training across 10 diverse base models, spanning different
  families and sizes including LLama3-8B, Mistral-7B/24B, DeepSeek-Math-7B,
  Qwen2.5-math-7B, and all Qwen2.5 models from 0.5B to 32B. Leveraging several
  key design strategies-such as adjusting format reward and controlling query
  difficulty-we achieve substantial improvements in both reasoning accuracy and
  response length across most settings. However, by carefully monitoring the
  training dynamics, we observe that different base models exhibit distinct
  patterns during training. For instance, the increased response length does not
  always correlate with the emergence of certain cognitive behaviors such as
  verification (i.e., the "aha moment"). Notably, we observe the "aha moment" for
  the first time in small models not from the Qwen family. We share the key
  designs that enable successful zero RL training, along with our findings and
  practices. To facilitate further research, we open-source the code, models, and
  analysis tools.
paper_authors._id: 67e22ce1155ea10f2fdbe5d2
paper_authors.name: Weihao Zeng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-25T08:20:18.367Z
paper_authors._id: 62751082b43ccfeef483424f
paper_authors.avatarUrl: /avatars/fec83e4478e7d1731ba6033328131852.svg
paper_authors.fullname: WeihaoZeng
paper_authors.type: user
paper_authors.user: AndrewZeng
paper_authors._id: 67e22ce1155ea10f2fdbe5d3
paper_authors.name: Yuzhen Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-25T08:20:13.781Z
paper_authors._id: 6462def82a83863b97c0611e
paper_authors.avatarUrl: /avatars/c03e9cc7d75b0266fcc56ecb6ee62148.svg
paper_authors.fullname: Yuzhen Huang
paper_authors.type: user
paper_authors.user: yuzhen17
paper_authors._id: 67e22ce1155ea10f2fdbe5d4
paper_authors.name: Qian Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-25T08:20:15.927Z
paper_authors._id: 612ee6a7b960e78c6d2319d4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/612ee6a7b960e78c6d2319d4/2Hu9BaAyXbyh1vt0v1Qui.jpeg
paper_authors.fullname: Qian Liu
paper_authors.type: user
paper_authors.user: SivilTaram
paper_authors._id: 67e22ce1155ea10f2fdbe5d5
paper_authors.name: Wei Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-22T07:18:37.813Z
paper_authors._id: 6458af46f4d212d780bd7c68
paper_authors.avatarUrl: /avatars/832fd34bcc041b0b7b551873a459fc3c.svg
paper_authors.fullname: Wei Liu
paper_authors.type: user
paper_authors.user: PeterV09
paper_authors._id: 67e22ce1155ea10f2fdbe5d6
paper_authors.name: Keqing He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-25T09:17:09.770Z
paper_authors._id: 64bf71792915a87970c07446
paper_authors.avatarUrl: /avatars/b24403f9fa699e0143e441b56528e6af.svg
paper_authors.fullname: Keqing He
paper_authors.type: user
paper_authors.user: HelicHe
paper_authors._id: 67e22ce1155ea10f2fdbe5d7
paper_authors.name: Zejun Ma
paper_authors._id: 67e22ce1155ea10f2fdbe5d8
paper_authors.name: Junxian He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-25T09:16:49.208Z
paper_authors._id: 615f34ec3f6d24d67c1b5c78
paper_authors.avatarUrl: /avatars/6dcff6477993d9e57c5cb92b6f95eb66.svg
paper_authors.fullname: Junxian He
paper_authors.type: user
paper_authors.user: jxhe
paper_title: SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for
    Open Base Models in the Wild
paper_summary: DeepSeek-R1 has shown that long chain-of-thought (CoT) reasoning can
  naturally emerge through a simple reinforcement learning (RL) framework with
  rule-based rewards, where the training may directly start from the base
  models-a paradigm referred to as zero RL training. Most recent efforts to
  reproduce zero RL training have primarily focused on the Qwen2.5 model series,
  which may not be representative as we find the base models already exhibit
  strong instruction-following and self-reflection abilities. In this work, we
  investigate zero RL training across 10 diverse base models, spanning different
  families and sizes including LLama3-8B, Mistral-7B/24B, DeepSeek-Math-7B,
  Qwen2.5-math-7B, and all Qwen2.5 models from 0.5B to 32B. Leveraging several
  key design strategies-such as adjusting format reward and controlling query
  difficulty-we achieve substantial improvements in both reasoning accuracy and
  response length across most settings. However, by carefully monitoring the
  training dynamics, we observe that different base models exhibit distinct
  patterns during training. For instance, the increased response length does not
  always correlate with the emergence of certain cognitive behaviors such as
  verification (i.e., the "aha moment"). Notably, we observe the "aha moment" for
  the first time in small models not from the Qwen family. We share the key
  designs that enable successful zero RL training, along with our findings and
  practices. To facilitate further research, we open-source the code, models, and
  analysis tools.
paper_ai_summary: Zero reinforcement learning training across diverse base models improves reasoning accuracy and response length through strategic design adjustments, revealing novel cognitive behaviors in smaller models.
paper_ai_keywords: reinforcement learning zero RL training long chain-of-thought reasoning rule-based rewards base models Qwen2.5 LLama3-8B Mistral-7B/24B DeepSeek-Math-7B Qwen2.5-math-7B format reward query difficulty reasoning accuracy response length cognitive behaviors "aha moment" open-source code analysis tools