Preference Ranking Optimization for Human Alignment
Large language models (LLMs) often contain misleading content, emphasizing
the need to align them with human values to ensure secur AI systems.
Reinforcement learning from human feedback (RLHF) has been employed to achieve
this alignment by combining a reward model, typically based on Bradley-Terry
paired comparison, with an RL algorithm such as Proximal Policy Optimization
(PPO) to optimize LLM responses. However, RLHF exhibits complexity,
instability, and sensitivity to hyperparameters. In this paper, we propose
Preference Ranking Optimization (PRO) as an alternative to PPO for directly
aligning LLMs with the Bradley-Terry comparison. PRO extends the pairwise
Bradley-Terry comparison to accommodate preference rankings of any length. By
iteratively contrasting the likelihood of generating responses, PRO instructs
the LLM to prioritize the best response while progressively ranking the
remaining responses. In this manner, PRO effectively transforms human alignment
into aligning the probability ranking of n responses generated by LLM with
the preference ranking of humans towards these responses. Experiments have
shown that PRO outperforms existing alignment algorithms, achieving comparable
results to ChatGPT and human responses through automatic-based, reward-based,
GPT-4, and human evaluations. Furthermore, we demonstrate that longer, more
diverse, and higher-quality preference ranking sequences can consistently
enhance the performance of human alignment.
64a21ed6a1a76b1d016dfc08
Feifan Song
claimed_verified
2023-07-03T15:22:57.495Z
6447ca6ca478b20f1755b294
/avatars/5049856b5ed1b74533fff902e14b4c7c.svg
Feifan Song
user
songff
64a21ed6a1a76b1d016dfc09
Bowen Yu
claimed_verified
2023-07-17T06:49:20.363Z
63d9d68c1cae35c27bf7a6a7
/avatars/b5ad98cf269ae5f1fe90861fb4170fae.svg
Bowen Yu
user
Tigerph
64a21ed6a1a76b1d016dfc0a
Minghao Li
64a21ed6a1a76b1d016dfc0b
Haiyang Yu
claimed_verified
2024-09-30T08:10:47.493Z
64a4ceda9a90f701134189b7
/avatars/859a189c5d2ae2fcb9aa2d79104fbfe7.svg
Haiyang Yu
user
yhycai
64a21ed6a1a76b1d016dfc0c
Fei Huang
64a21ed6a1a76b1d016dfc0d
Yongbin Li
claimed_verified
2024-10-14T10:48:24.193Z
66641b2fd8e1e34bc621e688
https://cdn-avatars.huggingface.co/v1/production/uploads/66641b2fd8e1e34bc621e688/csPETwnx2zCIHSWi9uAi-.png
Yongbin Li
user
Yongbin-Li
64a21ed6a1a76b1d016dfc0e
Houfeng Wang
Preference Ranking Optimization for Human Alignment
Large language models (LLMs) often contain misleading content, emphasizing
the need to align them with human values to ensure secur AI systems.
Reinforcement learning from human feedback (RLHF) has been employed to achieve
this alignment by combining a reward model, typically based on Bradley-Terry
paired comparison, with an RL algorithm such as Proximal Policy Optimization
(PPO) to optimize LLM responses. However, RLHF exhibits complexity,
instability, and sensitivity to hyperparameters. In this paper, we propose
Preference Ranking Optimization (PRO) as an alternative to PPO for directly
aligning LLMs with the Bradley-Terry comparison. PRO extends the pairwise
Bradley-Terry comparison to accommodate preference rankings of any length. By
iteratively contrasting the likelihood of generating responses, PRO instructs
the LLM to prioritize the best response while progressively ranking the
remaining responses. In this manner, PRO effectively transforms human alignment
into aligning the probability ranking of n responses generated by LLM with
the preference ranking of humans towards these responses. Experiments have
shown that PRO outperforms existing alignment algorithms, achieving comparable
results to ChatGPT and human responses through automatic-based, reward-based,
GPT-4, and human evaluations. Furthermore, we demonstrate that longer, more
diverse, and higher-quality preference ranking sequences can consistently
enhance the performance of human alignment.
A new method, Preference Ranking Optimization (PRO), is introduced to align LLMs with human preferences by extending the Bradley-Terry model to longer preference rankings, outperforming existing reinforcement learning techniques.
reinforcement learning from human feedback (RLHF) reward model Proximal Policy Optimization (PPO) Preference Ranking Optimization (PRO) Bradley-Terry paired comparison likelihood of generating responses human alignment preference ranking