title: WPO: Enhancing RLHF with Weighted Preference Optimization
summary: Reinforcement learning from human feedback (RLHF) is a promising solution to
  align large language models (LLMs) more closely with human values. Off-policy
  preference optimization, where the preference data is obtained from other
  models, is widely adopted due to its cost efficiency and scalability. However,
  off-policy preference optimization often suffers from a distributional gap
  between the policy used for data collection and the target policy, leading to
  suboptimal optimization. In this paper, we propose a novel strategy to mitigate
  this problem by simulating on-policy learning with off-policy preference data.
  Our Weighted Preference Optimization (WPO) method adapts off-policy data to
  resemble on-policy data more closely by reweighting preference pairs according
  to their probability under the current policy. This method not only addresses
  the distributional gap problem but also enhances the optimization process
  without incurring additional costs. We validate our method on instruction
  following benchmarks including Alpaca Eval 2 and MT-bench. WPO not only
  outperforms Direct Preference Optimization (DPO) by up to 5.6% on Alpaca Eval 2
  but also establishes a remarkable length-controlled winning rate against
  GPT-4-turbo of 48.6% based on Llama-3-8B-Instruct, making it the strongest 8B
  model on the leaderboard. We will release the code and models at
  https://github.com/wzhouad/WPO.
paper_authors._id: 667113ce307814981d58bb23
paper_authors.name: Wenxuan Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-18T08:19:09.321Z
paper_authors._id: 619414c7979eb76ec3cc5bab
paper_authors.avatarUrl: /avatars/c1a7055db6150b1bba92e7b13266b489.svg
paper_authors.fullname: Wenxuan Zhou
paper_authors.type: user
paper_authors.user: wzhouad
paper_authors._id: 667113ce307814981d58bb24
paper_authors.name: Ravi Agrawal
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-18T21:26:56.761Z
paper_authors._id: 62e96f0ae9dd82c094f48ac8
paper_authors.avatarUrl: /avatars/2ee9aced6c9341e51be0567e94e4eb8b.svg
paper_authors.fullname: Ravi Agrawal
paper_authors.type: user
paper_authors.user: ravi20036
paper_authors._id: 667113ce307814981d58bb25
paper_authors.name: Shujian Zhang
paper_authors._id: 667113ce307814981d58bb26
paper_authors.name: Sathish Reddy Indurthi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-18T15:59:43.662Z
paper_authors._id: 631059bea951093de05aedbd
paper_authors.avatarUrl: /avatars/f44ebb00a09776c29dd6ec96dcd91f24.svg
paper_authors.fullname: Sathish Reddy Indurthi
paper_authors.type: user
paper_authors.user: isathish
paper_authors._id: 667113ce307814981d58bb27
paper_authors.name: Sanqiang Zhao
paper_authors._id: 667113ce307814981d58bb28
paper_authors.name: Kaiqiang Song
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-18T15:59:27.216Z
paper_authors._id: 63a261f13c003e409322c191
paper_authors.avatarUrl: /avatars/dc4be6ce1983d0d32620275df124fe45.svg
paper_authors.fullname: Kaiqiang Song
paper_authors.type: user
paper_authors.user: kqsong
paper_authors._id: 667113ce307814981d58bb29
paper_authors.name: Silei Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-18T15:59:19.071Z
paper_authors._id: 6594b1bb57a556fbe162915e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6594b1bb57a556fbe162915e/WuYxqbbvaJaT-xsk5KhoT.jpeg
paper_authors.fullname: Silei Xu
paper_authors.type: user
paper_authors.user: sileixu
paper_authors._id: 667113ce307814981d58bb2a
paper_authors.name: Chenguang Zhu
paper_title: WPO: Enhancing RLHF with Weighted Preference Optimization
paper_summary: Reinforcement learning from human feedback (RLHF) is a promising solution to
  align large language models (LLMs) more closely with human values. Off-policy
  preference optimization, where the preference data is obtained from other
  models, is widely adopted due to its cost efficiency and scalability. However,
  off-policy preference optimization often suffers from a distributional gap
  between the policy used for data collection and the target policy, leading to
  suboptimal optimization. In this paper, we propose a novel strategy to mitigate
  this problem by simulating on-policy learning with off-policy preference data.
  Our Weighted Preference Optimization (WPO) method adapts off-policy data to
  resemble on-policy data more closely by reweighting preference pairs according
  to their probability under the current policy. This method not only addresses
  the distributional gap problem but also enhances the optimization process
  without incurring additional costs. We validate our method on instruction
  following benchmarks including Alpaca Eval 2 and MT-bench. WPO not only
  outperforms Direct Preference Optimization (DPO) by up to 5.6% on Alpaca Eval 2
  but also establishes a remarkable length-controlled winning rate against
  GPT-4-turbo of 48.6% based on Llama-3-8B-Instruct, making it the strongest 8B
  model on the leaderboard. We will release the code and models at
  https://github.com/wzhouad/WPO.
paper_ai_summary: A novel weighted preference optimization method addresses distributional gaps in reinforcement learning from human feedback, improving performance on instruction-following benchmarks.
paper_ai_keywords: reinforcement learning from human feedback large language models off-policy preference optimization distributional gap on-policy learning instruction following benchmarks Alpaca Eval 2 MT-bench Direct Preference Optimization GPT-4-turbo Llama-3-8B-Instruct