title: Accelerated Preference Optimization for Large Language Model Alignment
summary: Reinforcement Learning from Human Feedback (RLHF) has emerged as a pivotal
  tool for aligning large language models (LLMs) with human preferences. Direct
  Preference Optimization (DPO), one of the most popular approaches, formulates
  RLHF as a policy optimization problem without explicitly estimating the reward
  function. It overcomes the stability and efficiency issues of two-step
  approaches, which typically involve first estimating the reward function and
  then optimizing the policy via proximal policy optimization (PPO). Since RLHF
  is essentially an optimization problem, and it is well-known that momentum
  techniques can accelerate optimization both theoretically and empirically, a
  natural question arises: Can RLHF be accelerated by momentum? This paper
  answers this question in the affirmative. In detail, we first show that the
  iterative preference optimization method can be viewed as a proximal point
  method. Based on this observation, we propose a general Accelerated Preference
  Optimization (APO) framework, which unifies many existing preference
  optimization algorithms and employs Nesterov's momentum technique to speed up
  the alignment of LLMs. Theoretically, we demonstrate that APO can achieve a
  faster convergence rate than the standard iterative preference optimization
  methods, including DPO and Self-Play Preference Optimization (SPPO).
  Empirically, we show the superiority of APO over DPO, iterative DPO, and other
  strong baselines for RLHF on the AlpacaEval 2.0 benchmark.
paper_authors._id: 6709c45f705db29c00d0684a
paper_authors.name: Jiafan He
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-10-12T00:35:44.469Z
paper_authors._id: 668857a3c1449735756c214c
paper_authors.avatarUrl: /avatars/4d7b5eae5a0b8eedf3eac6577adbf493.svg
paper_authors.fullname: jiafan he
paper_authors.type: user
paper_authors.user: mcykin
paper_authors._id: 6709c45f705db29c00d0684b
paper_authors.name: Huizhuo Yuan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-18T20:58:18.495Z
paper_authors._id: 657be24e8d360b690d5b665f
paper_authors.avatarUrl: /avatars/423185c3f386e3fec2e5708c1030b013.svg
paper_authors.fullname: Angela Yuan
paper_authors.type: user
paper_authors.user: angelahzyuan
paper_authors._id: 6709c45f705db29c00d0684c
paper_authors.name: Quanquan Gu
paper_title: Accelerated Preference Optimization for Large Language Model Alignment
paper_summary: Reinforcement Learning from Human Feedback (RLHF) has emerged as a pivotal
  tool for aligning large language models (LLMs) with human preferences. Direct
  Preference Optimization (DPO), one of the most popular approaches, formulates
  RLHF as a policy optimization problem without explicitly estimating the reward
  function. It overcomes the stability and efficiency issues of two-step
  approaches, which typically involve first estimating the reward function and
  then optimizing the policy via proximal policy optimization (PPO). Since RLHF
  is essentially an optimization problem, and it is well-known that momentum
  techniques can accelerate optimization both theoretically and empirically, a
  natural question arises: Can RLHF be accelerated by momentum? This paper
  answers this question in the affirmative. In detail, we first show that the
  iterative preference optimization method can be viewed as a proximal point
  method. Based on this observation, we propose a general Accelerated Preference
  Optimization (APO) framework, which unifies many existing preference
  optimization algorithms and employs Nesterov's momentum technique to speed up
  the alignment of LLMs. Theoretically, we demonstrate that APO can achieve a
  faster convergence rate than the standard iterative preference optimization
  methods, including DPO and Self-Play Preference Optimization (SPPO).
  Empirically, we show the superiority of APO over DPO, iterative DPO, and other
  strong baselines for RLHF on the AlpacaEval 2.0 benchmark.
paper_ai_summary: Momentum techniques are utilized to accelerate Reinforcement Learning from Human Feedback, leading to faster convergence and improved performance over existing methods.
paper_ai_keywords: Reinforcement Learning from Human Feedback Direct Preference Optimization policy optimization proximal policy optimization preference optimization iterated preference optimization proximal point method Accelerated Preference Optimization Nesterov's momentum Self-Play Preference Optimization AlpacaEval 2.0 benchmark