Truncated Proximal Policy Optimization
Recently, test-time scaling Large Language Models (LLMs) have demonstrated
exceptional reasoning capabilities across scientific and professional tasks by
generating long chains-of-thought (CoT). As a crucial component for developing
these reasoning models, reinforcement learning (RL), exemplified by Proximal
Policy Optimization (PPO) and its variants, allows models to learn through
trial and error. However, PPO can be time-consuming due to its inherent
on-policy nature, which is further exacerbated by increasing response lengths.
In this work, we propose Truncated Proximal Policy Optimization (T-PPO), a
novel extension to PPO that improves training efficiency by streamlining policy
update and length-restricted response generation. T-PPO mitigates the issue of
low hardware utilization, an inherent drawback of fully synchronized
long-generation procedures, where resources often sit idle during the waiting
periods for complete rollouts. Our contributions are two-folds. First, we
propose Extended Generalized Advantage Estimation (EGAE) for advantage
estimation derived from incomplete responses while maintaining the integrity of
policy learning. Second, we devise a computationally optimized mechanism that
allows for the independent optimization of the policy and value models. By
selectively filtering prompt and truncated tokens, this mechanism reduces
redundant computations and accelerates the training process without sacrificing
convergence performance. We demonstrate the effectiveness and efficacy of T-PPO
on AIME 2024 with a 32B base model. The experimental results show that T-PPO
improves the training efficiency of reasoning LLMs by up to 2.5x and
outperforms its existing competitors.
68539c6199bf39f9665c79f6
Tiantian Fan
68539c6199bf39f9665c79f7
Lingjun Liu
68539c6199bf39f9665c79f8
Yu Yue
68539c6199bf39f9665c79f9
Jiaze Chen
68539c6199bf39f9665c79fa
Chengyi Wang
68539c6199bf39f9665c79fb
Qiying Yu
68539c6199bf39f9665c79fc
Chi Zhang
68539c6199bf39f9665c79fd
Zhiqi Lin
68539c6199bf39f9665c79fe
Ruofei Zhu
68539c6199bf39f9665c79ff
Yufeng Yuan
68539c6199bf39f9665c7a00
Xiaochen Zuo
68539c6199bf39f9665c7a01
Bole Ma
68539c6199bf39f9665c7a02
Mofan Zhang
68539c6199bf39f9665c7a03
Gaohong Liu
68539c6199bf39f9665c7a04
Ru Zhang
68539c6199bf39f9665c7a05
Haotian Zhou
68539c6199bf39f9665c7a06
Cong Xie
68539c6199bf39f9665c7a07
Ruidong Zhu
68539c6199bf39f9665c7a08
Zhi Zhang
68539c6199bf39f9665c7a09
Xin Liu
68539c6199bf39f9665c7a0a
Mingxuan Wang
68539c6199bf39f9665c7a0b
Lin Yan
68539c6199bf39f9665c7a0c
Yonghui Wu
Truncated Proximal Policy Optimization
Recently, test-time scaling Large Language Models (LLMs) have demonstrated
exceptional reasoning capabilities across scientific and professional tasks by
generating long chains-of-thought (CoT). As a crucial component for developing
these reasoning models, reinforcement learning (RL), exemplified by Proximal
Policy Optimization (PPO) and its variants, allows models to learn through
trial and error. However, PPO can be time-consuming due to its inherent
on-policy nature, which is further exacerbated by increasing response lengths.
In this work, we propose Truncated Proximal Policy Optimization (T-PPO), a
novel extension to PPO that improves training efficiency by streamlining policy
update and length-restricted response generation. T-PPO mitigates the issue of
low hardware utilization, an inherent drawback of fully synchronized
long-generation procedures, where resources often sit idle during the waiting
periods for complete rollouts. Our contributions are two-folds. First, we
propose Extended Generalized Advantage Estimation (EGAE) for advantage
estimation derived from incomplete responses while maintaining the integrity of
policy learning. Second, we devise a computationally optimized mechanism that
allows for the independent optimization of the policy and value models. By
selectively filtering prompt and truncated tokens, this mechanism reduces
redundant computations and accelerates the training process without sacrificing
convergence performance. We demonstrate the effectiveness and efficacy of T-PPO
on AIME 2024 with a 32B base model. The experimental results show that T-PPO
improves the training efficiency of reasoning LLMs by up to 2.5x and
outperforms its existing competitors.
T-PPO, an extension of PPO, improves training efficiency for Large Language Models by optimizing policy updates and utilizing hardware resources more effectively.
Large Language Models (LLMs) chains-of-thought (CoT) reinforcement learning (RL) Proximal Policy Optimization (PPO) Truncated Proximal Policy Optimization (T-PPO) Extended Generalized Advantage Estimation (EGAE) advantage estimation policy and value models independent optimization prompt and truncated tokens AIME 2024 base model