title: VAPO: Efficient and Reliable Reinforcement Learning for Advanced
    Reasoning Tasks
summary: We present VAPO, Value-based Augmented Proximal Policy Optimization framework
  for reasoning models., a novel framework tailored for reasoning models within
  the value-based paradigm. Benchmarked the AIME 2024 dataset, VAPO, built on the
  Qwen 32B pre-trained model, attains a state-of-the-art score of
  60.4. In direct comparison under identical experimental settings,
  VAPO outperforms the previously reported results of DeepSeek-R1-Zero-Qwen-32B
  and DAPO by more than 10 points. The training process of VAPO stands out for
  its stability and efficiency. It reaches state-of-the-art performance within a
  mere 5,000 steps. Moreover, across multiple independent runs, no training
  crashes occur, underscoring its reliability. This research delves into long
  chain-of-thought (long-CoT) reasoning using a value-based reinforcement
  learning framework. We pinpoint three key challenges that plague value-based
  methods: value model bias, the presence of heterogeneous sequence lengths, and
  the sparsity of reward signals. Through systematic design, VAPO offers an
  integrated solution that effectively alleviates these challenges, enabling
  enhanced performance in long-CoT reasoning tasks.
paper_authors._id: 67f503bdcb45555af74243e3
paper_authors.name: YuYue
paper_authors._id: 67f503bdcb45555af74243e4
paper_authors.name: Yufeng Yuan
paper_authors._id: 67f503bdcb45555af74243e5
paper_authors.name: Qiying Yu
paper_authors._id: 67f503bdcb45555af74243e6
paper_authors.name: Xiaochen Zuo
paper_authors._id: 67f503bdcb45555af74243e7
paper_authors.name: Ruofei Zhu
paper_authors._id: 67f503bdcb45555af74243e8
paper_authors.name: Wenyuan Xu
paper_authors._id: 67f503bdcb45555af74243e9
paper_authors.name: Jiaze Chen
paper_authors._id: 67f503bdcb45555af74243ea
paper_authors.name: Chengyi Wang
paper_authors._id: 67f503bdcb45555af74243eb
paper_authors.name: TianTian Fan
paper_authors._id: 67f503bdcb45555af74243ec
paper_authors.name: Zhengyin Du
paper_authors._id: 67f503bdcb45555af74243ed
paper_authors.name: Xiangpeng Wei
paper_authors._id: 67f503bdcb45555af74243ee
paper_authors.name: Gaohong Liu
paper_authors._id: 67f503bdcb45555af74243ef
paper_authors.name: Juncai Liu
paper_authors._id: 67f503bdcb45555af74243f0
paper_authors.name: Lingjun Liu
paper_authors._id: 67f503bdcb45555af74243f1
paper_authors.name: Haibin Lin
paper_authors._id: 67f503bdcb45555af74243f2
paper_authors.name: Zhiqi Lin
paper_authors._id: 67f503bdcb45555af74243f3
paper_authors.name: Bole Ma
paper_authors._id: 67f503bdcb45555af74243f4
paper_authors.name: Chi Zhang
paper_authors._id: 67f503bdcb45555af74243f5
paper_authors.name: Mofan Zhang
paper_authors._id: 67f503bdcb45555af74243f6
paper_authors.name: Wang Zhang
paper_authors._id: 67f503bdcb45555af74243f7
paper_authors.name: Hang Zhu
paper_authors._id: 67f503bdcb45555af74243f8
paper_authors.name: Ru Zhang
paper_authors._id: 67f503bdcb45555af74243f9
paper_authors.name: Xin Liu
paper_authors._id: 67f503bdcb45555af74243fa
paper_authors.name: Mingxuan Wang
paper_authors._id: 67f503bdcb45555af74243fb
paper_authors.name: Yonghui Wu
paper_authors._id: 67f503bdcb45555af74243fc
paper_authors.name: Lin Yan
paper_title: VAPO: Efficient and Reliable Reinforcement Learning for Advanced
    Reasoning Tasks
paper_summary: We present VAPO, Value-based Augmented Proximal Policy Optimization framework
  for reasoning models., a novel framework tailored for reasoning models within
  the value-based paradigm. Benchmarked the AIME 2024 dataset, VAPO, built on the
  Qwen 32B pre-trained model, attains a state-of-the-art score of
  60.4. In direct comparison under identical experimental settings,
  VAPO outperforms the previously reported results of DeepSeek-R1-Zero-Qwen-32B
  and DAPO by more than 10 points. The training process of VAPO stands out for
  its stability and efficiency. It reaches state-of-the-art performance within a
  mere 5,000 steps. Moreover, across multiple independent runs, no training
  crashes occur, underscoring its reliability. This research delves into long
  chain-of-thought (long-CoT) reasoning using a value-based reinforcement
  learning framework. We pinpoint three key challenges that plague value-based
  methods: value model bias, the presence of heterogeneous sequence lengths, and
  the sparsity of reward signals. Through systematic design, VAPO offers an
  integrated solution that effectively alleviates these challenges, enabling
  enhanced performance in long-CoT reasoning tasks.
paper_ai_summary: VAPO, a value-based augmented proximal policy optimization framework, achieves state-of-the-art performance on reasoning tasks using a value-based reinforcement learning approach.
paper_ai_keywords: Value-based Augmented Proximal Policy Optimization value-based paradigm Qwen 32B long chain-of-thought reasoning long-CoT reasoning value model bias heterogeneous sequence lengths sparsity of reward signals reinforcement learning