title: DAPO: An Open-Source LLM Reinforcement Learning System at Scale
summary: Inference scaling empowers LLMs with unprecedented reasoning ability, with
  reinforcement learning as the core technique to elicit complex reasoning.
  However, key technical details of state-of-the-art reasoning LLMs are concealed
  (such as in OpenAI o1 blog and DeepSeek R1 technical report), thus the
  community still struggles to reproduce their RL training results. We propose
  the Decoupled Clip and Dynamic sAmpling
  Policy Optimization (DAPO) algorithm, and
  fully open-source a state-of-the-art large-scale RL system that achieves 50
  points on AIME 2024 using Qwen2.5-32B base model. Unlike previous works that
  withhold training details, we introduce four key techniques of our algorithm
  that make large-scale LLM RL a success. In addition, we open-source our
  training code, which is built on the verl framework, along with a carefully
  curated and processed dataset. These components of our open-source system
  enhance reproducibility and support future research in large-scale LLM RL.
paper_authors._id: 67da2b54e5335651349e262c
paper_authors.name: Qiying Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-19T13:19:18.079Z
paper_authors._id: 6341375be5071b0c5cc946a3
paper_authors.avatarUrl: /avatars/c26a32b54a3f7731854059d247b5c523.svg
paper_authors.fullname: Qiying Yu
paper_authors.type: user
paper_authors.user: qiying
paper_authors._id: 67da2b54e5335651349e262d
paper_authors.name: Zheng Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-19T13:19:31.824Z
paper_authors._id: 648e9244e6598457da62a180
paper_authors.avatarUrl: /avatars/60329251f08e93260196ff67a842c188.svg
paper_authors.fullname: Zheng Zhang
paper_authors.type: user
paper_authors.user: zhengzhang
paper_authors._id: 67da2b54e5335651349e262e
paper_authors.name: Ruofei Zhu
paper_authors._id: 67da2b54e5335651349e262f
paper_authors.name: Yufeng Yuan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-19T13:19:54.321Z
paper_authors._id: 64b755e500bac1088cea02f1
paper_authors.avatarUrl: /avatars/f4adf70175e84e945436d1f091ba338b.svg
paper_authors.fullname: Yufeng Yuan
paper_authors.type: user
paper_authors.user: yufeng10
paper_authors._id: 67da2b54e5335651349e2630
paper_authors.name: Xiaochen Zuo
paper_authors._id: 67da2b54e5335651349e2631
paper_authors.name: Yu Yue
paper_authors._id: 67da2b54e5335651349e2632
paper_authors.name: Tiantian Fan
paper_authors._id: 67da2b54e5335651349e2633
paper_authors.name: Gaohong Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-19T13:20:22.957Z
paper_authors._id: 61512f1b3e89795099b6fced
paper_authors.avatarUrl: /avatars/8f19253df7343e9d95ad7d0b3d429c06.svg
paper_authors.fullname: Gaohong Liu
paper_authors.type: user
paper_authors.user: Cheimu
paper_authors._id: 67da2b54e5335651349e2634
paper_authors.name: Lingjun Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-19T13:20:29.992Z
paper_authors._id: 6626274011772517e5469d48
paper_authors.avatarUrl: /avatars/5f2606614740da04a7fb9d80fa628fb1.svg
paper_authors.fullname: Lingjun Liu
paper_authors.type: user
paper_authors.user: EEchollj
paper_authors._id: 67da2b54e5335651349e2635
paper_authors.name: Xin Liu
paper_authors._id: 67da2b54e5335651349e2636
paper_authors.name: Haibin Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-19T13:20:36.744Z
paper_authors._id: 65e809f67c1853c8c180baff
paper_authors.avatarUrl: /avatars/e489eaebbbf86f8826fe0e9d5423be01.svg
paper_authors.fullname: haibin
paper_authors.type: user
paper_authors.user: haibinlin
paper_authors._id: 67da2b54e5335651349e2637
paper_authors.name: Zhiqi Lin
paper_authors._id: 67da2b54e5335651349e2638
paper_authors.name: Bole Ma
paper_authors._id: 67da2b54e5335651349e2639
paper_authors.name: Guangming Sheng
paper_authors._id: 67da2b54e5335651349e263a
paper_authors.name: Yuxuan Tong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-19T13:21:29.044Z
paper_authors._id: 6448e1fbe988635a3d6aa97d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/eG4R9-3hgrimttP7ep3dN.jpeg
paper_authors.fullname: Shawn/Yuxuan Tong
paper_authors.type: user
paper_authors.user: tongyx361
paper_authors._id: 67da2b54e5335651349e263b
paper_authors.name: Chi Zhang
paper_authors._id: 67da2b54e5335651349e263c
paper_authors.name: Mofan Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-19T13:21:51.391Z
paper_authors._id: 64f33235bbf2fd1b4816632b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/sVN-M1Hj-aB4QqytdpUsn.jpeg
paper_authors.fullname: Mofan Zhang
paper_authors.type: user
paper_authors.user: NewMomo
paper_authors._id: 67da2b54e5335651349e263d
paper_authors.name: Wang Zhang
paper_authors._id: 67da2b54e5335651349e263e
paper_authors.name: Hang Zhu
paper_authors._id: 67da2b54e5335651349e263f
paper_authors.name: Jinhua Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-19T13:22:23.684Z
paper_authors._id: 650b0009bddcb62e458ebadf
paper_authors.avatarUrl: /avatars/0431a38b9c9ca0e11ec505ce81183be8.svg
paper_authors.fullname: Jinhua Zhu
paper_authors.type: user
paper_authors.user: teslazhu
paper_authors._id: 67da2b54e5335651349e2640
paper_authors.name: Jiaze Chen
paper_authors._id: 67da2b54e5335651349e2641
paper_authors.name: Jiangjie Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-27T07:56:06.181Z
paper_authors._id: 606ed1884ffe81d1e03e81e5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1639375346654-606ed1884ffe81d1e03e81e5.png
paper_authors.fullname: Jiangjie Chen
paper_authors.type: user
paper_authors.user: jiangjiechen
paper_authors._id: 67da2b54e5335651349e2642
paper_authors.name: Chengyi Wang
paper_authors._id: 67da2b54e5335651349e2643
paper_authors.name: Hongli Yu
paper_authors._id: 67da2b54e5335651349e2644
paper_authors.name: Weinan Dai
paper_authors._id: 67da2b54e5335651349e2645
paper_authors.name: Yuxuan Song
paper_authors._id: 67da2b54e5335651349e2646
paper_authors.name: Xiangpeng Wei
paper_authors._id: 67da2b54e5335651349e2647
paper_authors.name: Hao Zhou
paper_authors._id: 67da2b54e5335651349e2648
paper_authors.name: Jingjing Liu
paper_authors._id: 67da2b54e5335651349e2649
paper_authors.name: Wei-Ying Ma
paper_authors._id: 67da2b54e5335651349e264a
paper_authors.name: Ya-Qin Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-19T13:22:33.159Z
paper_authors._id: 656c1600665d15428a8bdc60
paper_authors.avatarUrl: /avatars/dfdf5f8712c769896b4ab655c4ce32e4.svg
paper_authors.fullname: Ya-Qin Zhang
paper_authors.type: user
paper_authors.user: Ya-Qin
paper_authors._id: 67da2b54e5335651349e264b
paper_authors.name: Lin Yan
paper_authors._id: 67da2b54e5335651349e264c
paper_authors.name: Mu Qiao
paper_authors._id: 67da2b54e5335651349e264d
paper_authors.name: Yonghui Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-19T13:22:16.717Z
paper_authors._id: 647a13ba5f3450e1ded4ca85
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/647a13ba5f3450e1ded4ca85/0ryiqG2gpFlffOFNq3jpj.jpeg
paper_authors.fullname: Yonghui Wu
paper_authors.type: user
paper_authors.user: yonghuiwu
paper_authors._id: 67da2b54e5335651349e264e
paper_authors.name: Mingxuan Wang
paper_title: DAPO: An Open-Source LLM Reinforcement Learning System at Scale
paper_summary: Inference scaling empowers LLMs with unprecedented reasoning ability, with
  reinforcement learning as the core technique to elicit complex reasoning.
  However, key technical details of state-of-the-art reasoning LLMs are concealed
  (such as in OpenAI o1 blog and DeepSeek R1 technical report), thus the
  community still struggles to reproduce their RL training results. We propose
  the Decoupled Clip and Dynamic sAmpling
  Policy Optimization (DAPO) algorithm, and
  fully open-source a state-of-the-art large-scale RL system that achieves 50
  points on AIME 2024 using Qwen2.5-32B base model. Unlike previous works that
  withhold training details, we introduce four key techniques of our algorithm
  that make large-scale LLM RL a success. In addition, we open-source our
  training code, which is built on the verl framework, along with a carefully
  curated and processed dataset. These components of our open-source system
  enhance reproducibility and support future research in large-scale LLM RL.
paper_ai_summary: The DAPO algorithm, which includes decoupled clip and dynamic sampling policy optimization, enables open-source, high-performance reinforcement learning training for large-scale language models, enhancing reproducibility in the field.
paper_ai_keywords: LLMs reinforcement learning decoupled clip dynamic sampling policy optimization DAPO large-scale RL open-source AIME 2024 Qwen2.5-32B verl framework