title: An Empirical Study on Eliciting and Improving R1-like Reasoning Models
summary: In this report, we present the third technical report on the development of
  slow-thinking models as part of the STILL project. As the technical pathway
  becomes clearer, scaling RL training has become a central technique for
  implementing such reasoning models. We systematically experiment with and
  document the effects of various factors influencing RL training, conducting
  experiments on both base models and fine-tuned models. Specifically, we
  demonstrate that our RL training approach consistently improves the Qwen2.5-32B
  base models, enhancing both response length and test accuracy. Furthermore, we
  show that even when a model like DeepSeek-R1-Distill-Qwen-1.5B has already
  achieved a high performance level, it can be further refined through RL
  training, reaching an accuracy of 39.33% on AIME 2024. Beyond RL training, we
  also explore the use of tool manipulation, finding that it significantly boosts
  the reasoning performance of large reasoning models. This approach achieves a
  remarkable accuracy of 86.67% with greedy search on AIME 2024, underscoring its
  effectiveness in enhancing model capabilities. We release our resources at the
  STILL project website: https://github.com/RUCAIBox/Slow_Thinking_with_LLMs.
paper_authors._id: 67cbff8e4dedec48bdec8a99
paper_authors.name: Zhipeng Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-10T11:08:52.643Z
paper_authors._id: 629b765ce1af194c641fcbc6
paper_authors.avatarUrl: /avatars/7c53a4c2a1e528c19641a2b601731754.svg
paper_authors.fullname: Zhipeng Chen
paper_authors.type: user
paper_authors.user: TimothyCzp
paper_authors._id: 67cbff8e4dedec48bdec8a9a
paper_authors.name: Yingqian Min
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-10T08:02:04.349Z
paper_authors._id: 6703ac76ea890f0ca5b225eb
paper_authors.avatarUrl: /avatars/5f56c49a1940143d47dd484782a4abbf.svg
paper_authors.fullname: Yingqian Min
paper_authors.type: user
paper_authors.user: EliverQ
paper_authors._id: 67cbff8e4dedec48bdec8a9b
paper_authors.name: Beichen Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-10T10:29:55.450Z
paper_authors._id: 648e6a4567aa8ab0e0e4c30f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/648e6a4567aa8ab0e0e4c30f/b9OsE6C0a5iJE1yAlFG-_.jpeg
paper_authors.fullname: Beichen Zhang
paper_authors.type: user
paper_authors.user: ToheartZhang
paper_authors._id: 67cbff8e4dedec48bdec8a9c
paper_authors.name: Jie Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-10T12:46:30.961Z
paper_authors._id: 651a29d566e78720a78317ec
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/651a29d566e78720a78317ec/icoHA_v1RPVfwTooN3fJb.png
paper_authors.fullname: Jie Chen
paper_authors.type: user
paper_authors.user: survivi
paper_authors._id: 67cbff8e4dedec48bdec8a9d
paper_authors.name: Jinhao Jiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-10T10:30:03.904Z
paper_authors._id: 61b8405b516a20acdf3b85ff
paper_authors.avatarUrl: /avatars/3d2eae7c163a80b73260087b05a4230b.svg
paper_authors.fullname: Jinhao Jiang
paper_authors.type: user
paper_authors.user: Boru
paper_authors._id: 67cbff8e4dedec48bdec8a9e
paper_authors.name: Daixuan Cheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-10T10:30:09.972Z
paper_authors._id: 649e6761f9134a06ed1e0cea
paper_authors.avatarUrl: /avatars/00b5dcb744c54a4aa18fe08efd70d6ff.svg
paper_authors.fullname: Daixuan Cheng
paper_authors.type: user
paper_authors.user: daixuancheng
paper_authors._id: 67cbff8e4dedec48bdec8a9f
paper_authors.name: Wayne Xin Zhao
paper_authors._id: 67cbff8e4dedec48bdec8aa0
paper_authors.name: Zheng Liu
paper_authors._id: 67cbff8e4dedec48bdec8aa1
paper_authors.name: Xu Miao
paper_authors._id: 67cbff8e4dedec48bdec8aa2
paper_authors.name: Yang Lu
paper_authors._id: 67cbff8e4dedec48bdec8aa3
paper_authors.name: Lei Fang
paper_authors._id: 67cbff8e4dedec48bdec8aa4
paper_authors.name: Zhongyuan Wang
paper_authors._id: 67cbff8e4dedec48bdec8aa5
paper_authors.name: Ji-Rong Wen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-10T10:30:44.799Z
paper_authors._id: 64b8c89052b7353d8c6a1013
paper_authors.avatarUrl: /avatars/cd59fffe81f6b07b4519540b8ff3d95f.svg
paper_authors.fullname: Ji-Rong Wen
paper_authors.type: user
paper_authors.user: jrwen
paper_title: An Empirical Study on Eliciting and Improving R1-like Reasoning Models
paper_summary: In this report, we present the third technical report on the development of
  slow-thinking models as part of the STILL project. As the technical pathway
  becomes clearer, scaling RL training has become a central technique for
  implementing such reasoning models. We systematically experiment with and
  document the effects of various factors influencing RL training, conducting
  experiments on both base models and fine-tuned models. Specifically, we
  demonstrate that our RL training approach consistently improves the Qwen2.5-32B
  base models, enhancing both response length and test accuracy. Furthermore, we
  show that even when a model like DeepSeek-R1-Distill-Qwen-1.5B has already
  achieved a high performance level, it can be further refined through RL
  training, reaching an accuracy of 39.33% on AIME 2024. Beyond RL training, we
  also explore the use of tool manipulation, finding that it significantly boosts
  the reasoning performance of large reasoning models. This approach achieves a
  remarkable accuracy of 86.67% with greedy search on AIME 2024, underscoring its
  effectiveness in enhancing model capabilities. We release our resources at the
  STILL project website: https://github.com/RUCAIBox/Slow_Thinking_with_LLMs.
paper_ai_keywords: RL training Qwen2.5-32B DeepSeek-R1-Distill-Qwen-1.5B AIME 2024 tool manipulation