WPO: Enhancing RLHF with Weighted Preference Optimization
Reinforcement learning from human feedback (RLHF) is a promising solution to
align large language models (LLMs) more closely with human values. Off-policy
preference optimization, where the preference data is obtained from other
models, is widely adopted due to its cost efficiency and scalability. However,
off-policy preference optimization often suffers from a distributional gap
between the policy used for data collection and the target policy, leading to
suboptimal optimization. In this paper, we propose a novel strategy to mitigate
this problem by simulating on-policy learning with off-policy preference data.
Our Weighted Preference Optimization (WPO) method adapts off-policy data to
resemble on-policy data more closely by reweighting preference pairs according
to their probability under the current policy. This method not only addresses
the distributional gap problem but also enhances the optimization process
without incurring additional costs. We validate our method on instruction
following benchmarks including Alpaca Eval 2 and MT-bench. WPO not only
outperforms Direct Preference Optimization (DPO) by up to 5.6% on Alpaca Eval 2
but also establishes a remarkable length-controlled winning rate against
GPT-4-turbo of 48.6% based on Llama-3-8B-Instruct, making it the strongest 8B
model on the leaderboard. We will release the code and models at
https://github.com/wzhouad/WPO.
667113ce307814981d58bb23
Wenxuan Zhou
admin_assigned
2024-06-18T08:19:09.321Z
619414c7979eb76ec3cc5bab
/avatars/c1a7055db6150b1bba92e7b13266b489.svg
Wenxuan Zhou
user
wzhouad
667113ce307814981d58bb24
Ravi Agrawal
claimed_verified
2024-06-18T21:26:56.761Z
62e96f0ae9dd82c094f48ac8
/avatars/2ee9aced6c9341e51be0567e94e4eb8b.svg
Ravi Agrawal
user
ravi20036
667113ce307814981d58bb25
Shujian Zhang
667113ce307814981d58bb26
Sathish Reddy Indurthi
admin_assigned
2024-06-18T15:59:43.662Z
631059bea951093de05aedbd
/avatars/f44ebb00a09776c29dd6ec96dcd91f24.svg
Sathish Reddy Indurthi
user
isathish
667113ce307814981d58bb27
Sanqiang Zhao
667113ce307814981d58bb28
Kaiqiang Song
admin_assigned
2024-06-18T15:59:27.216Z
63a261f13c003e409322c191
/avatars/dc4be6ce1983d0d32620275df124fe45.svg
Kaiqiang Song
user
kqsong
667113ce307814981d58bb29
Silei Xu
admin_assigned
2024-06-18T15:59:19.071Z
6594b1bb57a556fbe162915e
https://cdn-avatars.huggingface.co/v1/production/uploads/6594b1bb57a556fbe162915e/WuYxqbbvaJaT-xsk5KhoT.jpeg
Silei Xu
user
sileixu
667113ce307814981d58bb2a
Chenguang Zhu
WPO: Enhancing RLHF with Weighted Preference Optimization
Reinforcement learning from human feedback (RLHF) is a promising solution to
align large language models (LLMs) more closely with human values. Off-policy
preference optimization, where the preference data is obtained from other
models, is widely adopted due to its cost efficiency and scalability. However,
off-policy preference optimization often suffers from a distributional gap
between the policy used for data collection and the target policy, leading to
suboptimal optimization. In this paper, we propose a novel strategy to mitigate
this problem by simulating on-policy learning with off-policy preference data.
Our Weighted Preference Optimization (WPO) method adapts off-policy data to
resemble on-policy data more closely by reweighting preference pairs according
to their probability under the current policy. This method not only addresses
the distributional gap problem but also enhances the optimization process
without incurring additional costs. We validate our method on instruction
following benchmarks including Alpaca Eval 2 and MT-bench. WPO not only
outperforms Direct Preference Optimization (DPO) by up to 5.6% on Alpaca Eval 2
but also establishes a remarkable length-controlled winning rate against
GPT-4-turbo of 48.6% based on Llama-3-8B-Instruct, making it the strongest 8B
model on the leaderboard. We will release the code and models at
https://github.com/wzhouad/WPO.
A novel weighted preference optimization method addresses distributional gaps in reinforcement learning from human feedback, improving performance on instruction-following benchmarks.
reinforcement learning from human feedback large language models off-policy preference optimization distributional gap on-policy learning instruction following benchmarks Alpaca Eval 2 MT-bench Direct Preference Optimization GPT-4-turbo Llama-3-8B-Instruct