Accelerated Preference Optimization for Large Language Model Alignment
Reinforcement Learning from Human Feedback (RLHF) has emerged as a pivotal
tool for aligning large language models (LLMs) with human preferences. Direct
Preference Optimization (DPO), one of the most popular approaches, formulates
RLHF as a policy optimization problem without explicitly estimating the reward
function. It overcomes the stability and efficiency issues of two-step
approaches, which typically involve first estimating the reward function and
then optimizing the policy via proximal policy optimization (PPO). Since RLHF
is essentially an optimization problem, and it is well-known that momentum
techniques can accelerate optimization both theoretically and empirically, a
natural question arises: Can RLHF be accelerated by momentum? This paper
answers this question in the affirmative. In detail, we first show that the
iterative preference optimization method can be viewed as a proximal point
method. Based on this observation, we propose a general Accelerated Preference
Optimization (APO) framework, which unifies many existing preference
optimization algorithms and employs Nesterov's momentum technique to speed up
the alignment of LLMs. Theoretically, we demonstrate that APO can achieve a
faster convergence rate than the standard iterative preference optimization
methods, including DPO and Self-Play Preference Optimization (SPPO).
Empirically, we show the superiority of APO over DPO, iterative DPO, and other
strong baselines for RLHF on the AlpacaEval 2.0 benchmark.
6709c45f705db29c00d0684a
Jiafan He
extracted_pending
2024-10-12T00:35:44.469Z
668857a3c1449735756c214c
/avatars/4d7b5eae5a0b8eedf3eac6577adbf493.svg
jiafan he
user
mcykin
6709c45f705db29c00d0684b
Huizhuo Yuan
claimed_verified
2024-11-18T20:58:18.495Z
657be24e8d360b690d5b665f
/avatars/423185c3f386e3fec2e5708c1030b013.svg
Angela Yuan
user
angelahzyuan
6709c45f705db29c00d0684c
Quanquan Gu
Accelerated Preference Optimization for Large Language Model Alignment
Reinforcement Learning from Human Feedback (RLHF) has emerged as a pivotal
tool for aligning large language models (LLMs) with human preferences. Direct
Preference Optimization (DPO), one of the most popular approaches, formulates
RLHF as a policy optimization problem without explicitly estimating the reward
function. It overcomes the stability and efficiency issues of two-step
approaches, which typically involve first estimating the reward function and
then optimizing the policy via proximal policy optimization (PPO). Since RLHF
is essentially an optimization problem, and it is well-known that momentum
techniques can accelerate optimization both theoretically and empirically, a
natural question arises: Can RLHF be accelerated by momentum? This paper
answers this question in the affirmative. In detail, we first show that the
iterative preference optimization method can be viewed as a proximal point
method. Based on this observation, we propose a general Accelerated Preference
Optimization (APO) framework, which unifies many existing preference
optimization algorithms and employs Nesterov's momentum technique to speed up
the alignment of LLMs. Theoretically, we demonstrate that APO can achieve a
faster convergence rate than the standard iterative preference optimization
methods, including DPO and Self-Play Preference Optimization (SPPO).
Empirically, we show the superiority of APO over DPO, iterative DPO, and other
strong baselines for RLHF on the AlpacaEval 2.0 benchmark.
Momentum techniques are utilized to accelerate Reinforcement Learning from Human Feedback, leading to faster convergence and improved performance over existing methods.
Reinforcement Learning from Human Feedback Direct Preference Optimization policy optimization proximal policy optimization preference optimization iterated preference optimization proximal point method Accelerated Preference Optimization Nesterov's momentum Self-Play Preference Optimization AlpacaEval 2.0 benchmark