title: Preference Ranking Optimization for Human Alignment
summary: Large language models (LLMs) often contain misleading content, emphasizing
  the need to align them with human values to ensure secur AI systems.
  Reinforcement learning from human feedback (RLHF) has been employed to achieve
  this alignment by combining a reward model, typically based on Bradley-Terry
  paired comparison, with an RL algorithm such as Proximal Policy Optimization
  (PPO) to optimize LLM responses. However, RLHF exhibits complexity,
  instability, and sensitivity to hyperparameters. In this paper, we propose
  Preference Ranking Optimization (PRO) as an alternative to PPO for directly
  aligning LLMs with the Bradley-Terry comparison. PRO extends the pairwise
  Bradley-Terry comparison to accommodate preference rankings of any length. By
  iteratively contrasting the likelihood of generating responses, PRO instructs
  the LLM to prioritize the best response while progressively ranking the
  remaining responses. In this manner, PRO effectively transforms human alignment
  into aligning the probability ranking of n responses generated by LLM with
  the preference ranking of humans towards these responses. Experiments have
  shown that PRO outperforms existing alignment algorithms, achieving comparable
  results to ChatGPT and human responses through automatic-based, reward-based,
  GPT-4, and human evaluations. Furthermore, we demonstrate that longer, more
  diverse, and higher-quality preference ranking sequences can consistently
  enhance the performance of human alignment.
paper_authors._id: 64a21ed6a1a76b1d016dfc08
paper_authors.name: Feifan Song
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-03T15:22:57.495Z
paper_authors._id: 6447ca6ca478b20f1755b294
paper_authors.avatarUrl: /avatars/5049856b5ed1b74533fff902e14b4c7c.svg
paper_authors.fullname: Feifan Song
paper_authors.type: user
paper_authors.user: songff
paper_authors._id: 64a21ed6a1a76b1d016dfc09
paper_authors.name: Bowen Yu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-17T06:49:20.363Z
paper_authors._id: 63d9d68c1cae35c27bf7a6a7
paper_authors.avatarUrl: /avatars/b5ad98cf269ae5f1fe90861fb4170fae.svg
paper_authors.fullname: Bowen Yu
paper_authors.type: user
paper_authors.user: Tigerph
paper_authors._id: 64a21ed6a1a76b1d016dfc0a
paper_authors.name: Minghao Li
paper_authors._id: 64a21ed6a1a76b1d016dfc0b
paper_authors.name: Haiyang Yu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-30T08:10:47.493Z
paper_authors._id: 64a4ceda9a90f701134189b7
paper_authors.avatarUrl: /avatars/859a189c5d2ae2fcb9aa2d79104fbfe7.svg
paper_authors.fullname: Haiyang Yu
paper_authors.type: user
paper_authors.user: yhycai
paper_authors._id: 64a21ed6a1a76b1d016dfc0c
paper_authors.name: Fei Huang
paper_authors._id: 64a21ed6a1a76b1d016dfc0d
paper_authors.name: Yongbin Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-14T10:48:24.193Z
paper_authors._id: 66641b2fd8e1e34bc621e688
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/66641b2fd8e1e34bc621e688/csPETwnx2zCIHSWi9uAi-.png
paper_authors.fullname: Yongbin Li
paper_authors.type: user
paper_authors.user: Yongbin-Li
paper_authors._id: 64a21ed6a1a76b1d016dfc0e
paper_authors.name: Houfeng Wang
paper_title: Preference Ranking Optimization for Human Alignment
paper_summary: Large language models (LLMs) often contain misleading content, emphasizing
  the need to align them with human values to ensure secur AI systems.
  Reinforcement learning from human feedback (RLHF) has been employed to achieve
  this alignment by combining a reward model, typically based on Bradley-Terry
  paired comparison, with an RL algorithm such as Proximal Policy Optimization
  (PPO) to optimize LLM responses. However, RLHF exhibits complexity,
  instability, and sensitivity to hyperparameters. In this paper, we propose
  Preference Ranking Optimization (PRO) as an alternative to PPO for directly
  aligning LLMs with the Bradley-Terry comparison. PRO extends the pairwise
  Bradley-Terry comparison to accommodate preference rankings of any length. By
  iteratively contrasting the likelihood of generating responses, PRO instructs
  the LLM to prioritize the best response while progressively ranking the
  remaining responses. In this manner, PRO effectively transforms human alignment
  into aligning the probability ranking of n responses generated by LLM with
  the preference ranking of humans towards these responses. Experiments have
  shown that PRO outperforms existing alignment algorithms, achieving comparable
  results to ChatGPT and human responses through automatic-based, reward-based,
  GPT-4, and human evaluations. Furthermore, we demonstrate that longer, more
  diverse, and higher-quality preference ranking sequences can consistently
  enhance the performance of human alignment.
paper_ai_keywords: Reinforcement learning from human feedback (RLHF) reward model Bradley-Terry paired comparison Proximal Policy Optimization (PPO) Preference Ranking Optimization (PRO) likelihood probability ranking automatic-based reward-based GPT-4