An Empirical Study on Eliciting and Improving R1-like Reasoning Models
In this report, we present the third technical report on the development of
slow-thinking models as part of the STILL project. As the technical pathway
becomes clearer, scaling RL training has become a central technique for
implementing such reasoning models. We systematically experiment with and
document the effects of various factors influencing RL training, conducting
experiments on both base models and fine-tuned models. Specifically, we
demonstrate that our RL training approach consistently improves the Qwen2.5-32B
base models, enhancing both response length and test accuracy. Furthermore, we
show that even when a model like DeepSeek-R1-Distill-Qwen-1.5B has already
achieved a high performance level, it can be further refined through RL
training, reaching an accuracy of 39.33% on AIME 2024. Beyond RL training, we
also explore the use of tool manipulation, finding that it significantly boosts
the reasoning performance of large reasoning models. This approach achieves a
remarkable accuracy of 86.67% with greedy search on AIME 2024, underscoring its
effectiveness in enhancing model capabilities. We release our resources at the
STILL project website: https://github.com/RUCAIBox/Slow_Thinking_with_LLMs.
67cbff8e4dedec48bdec8a99
Zhipeng Chen
claimed_verified
2025-03-10T11:08:52.643Z
629b765ce1af194c641fcbc6
/avatars/7c53a4c2a1e528c19641a2b601731754.svg
Zhipeng Chen
user
TimothyCzp
67cbff8e4dedec48bdec8a9a
Yingqian Min
claimed_verified
2025-03-10T08:02:04.349Z
6703ac76ea890f0ca5b225eb
/avatars/5f56c49a1940143d47dd484782a4abbf.svg
Yingqian Min
user
EliverQ
67cbff8e4dedec48bdec8a9b
Beichen Zhang
admin_assigned
2025-03-10T10:29:55.450Z
648e6a4567aa8ab0e0e4c30f
https://cdn-avatars.huggingface.co/v1/production/uploads/648e6a4567aa8ab0e0e4c30f/b9OsE6C0a5iJE1yAlFG-_.jpeg
Beichen Zhang
user
ToheartZhang
67cbff8e4dedec48bdec8a9c
Jie Chen
claimed_verified
2025-03-10T12:46:30.961Z
651a29d566e78720a78317ec
https://cdn-avatars.huggingface.co/v1/production/uploads/651a29d566e78720a78317ec/8yTNvYeUs8cZo5GWvPPOJ.jpeg
Jie Chen
user
survivi
67cbff8e4dedec48bdec8a9d
Jinhao Jiang
admin_assigned
2025-03-10T10:30:03.904Z
61b8405b516a20acdf3b85ff
/avatars/3d2eae7c163a80b73260087b05a4230b.svg
Jinhao Jiang
user
Boru
67cbff8e4dedec48bdec8a9e
Daixuan Cheng
admin_assigned
2025-03-10T10:30:09.972Z
649e6761f9134a06ed1e0cea
https://cdn-avatars.huggingface.co/v1/production/uploads/649e6761f9134a06ed1e0cea/XNeKceE8xSwI0xWwWUwwJ.jpeg
Daixuan Cheng
user
daixuancheng
67cbff8e4dedec48bdec8a9f
Wayne Xin Zhao
67cbff8e4dedec48bdec8aa0
Zheng Liu
67cbff8e4dedec48bdec8aa1
Xu Miao
67cbff8e4dedec48bdec8aa2
Yang Lu
67cbff8e4dedec48bdec8aa3
Lei Fang
67cbff8e4dedec48bdec8aa4
Zhongyuan Wang
67cbff8e4dedec48bdec8aa5
Ji-Rong Wen
admin_assigned
2025-03-10T10:30:44.799Z
64b8c89052b7353d8c6a1013
/avatars/cd59fffe81f6b07b4519540b8ff3d95f.svg
Ji-Rong Wen
user
jrwen
An Empirical Study on Eliciting and Improving R1-like Reasoning Models
In this report, we present the third technical report on the development of
slow-thinking models as part of the STILL project. As the technical pathway
becomes clearer, scaling RL training has become a central technique for
implementing such reasoning models. We systematically experiment with and
document the effects of various factors influencing RL training, conducting
experiments on both base models and fine-tuned models. Specifically, we
demonstrate that our RL training approach consistently improves the Qwen2.5-32B
base models, enhancing both response length and test accuracy. Furthermore, we
show that even when a model like DeepSeek-R1-Distill-Qwen-1.5B has already
achieved a high performance level, it can be further refined through RL
training, reaching an accuracy of 39.33% on AIME 2024. Beyond RL training, we
also explore the use of tool manipulation, finding that it significantly boosts
the reasoning performance of large reasoning models. This approach achieves a
remarkable accuracy of 86.67% with greedy search on AIME 2024, underscoring its
effectiveness in enhancing model capabilities. We release our resources at the
STILL project website: https://github.com/RUCAIBox/Slow_Thinking_with_LLMs.
RL training and tool manipulation enhance the reasoning performance and accuracy of large language models, as demonstrated through experiments with Qwen2.5-32B and DeepSeek-R1-Distill-Qwen-1.5B on AIME 2024.
scaling RL training Qwen2.5-32B DeepSeek-R1-Distill-Qwen-1.5B response length test accuracy tool manipulation AIME 2024 greedy search