title: General Preference Modeling with Preference Representations for Aligning
    Language Models
summary: Modeling human preferences is crucial for aligning foundation models with
  human values. Traditional reward modeling methods, such as the Bradley-Terry
  (BT) reward model, fall short in expressiveness, particularly in addressing
  intransitive preferences. Although supervised pair preference models (PairPM)
  can express general preferences, their implementation is highly ad-hoc and
  cannot guarantee a consistent preference probability of compared pairs.
  Additionally, they impose high computational costs due to their quadratic query
  complexity when comparing multiple responses. In this paper, we introduce
  preference representation learning, an approach that embeds responses into a
  latent space to capture intricate preference structures efficiently, achieving
  linear query complexity. Additionally, we propose preference score-based
  General Preference Optimization (GPO), which generalizes reward-based
  reinforcement learning from human feedback. Experimental results show that our
  General Preference representation model (GPM) outperforms the BT reward model
  on the RewardBench benchmark with a margin of up to 5.6% and effectively models
  cyclic preferences where any BT reward model behaves like a random guess.
  Furthermore, evaluations on downstream tasks such as AlpacaEval2.0 and
  MT-Bench, following the language model post-training with GPO and our general
  preference model, reveal substantial performance improvements with margins up
  to 9.3%. These findings indicate that our method may enhance the alignment of
  foundation models with nuanced human values. The code is available at
  https://github.com/general-preference/general-preference-model.
paper_authors._id: 66ff4115162623b07c7434e3
paper_authors.name: Yifan Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-04T07:31:15.090Z
paper_authors._id: 647bf082aba7062fe5c51ca9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/647bf082aba7062fe5c51ca9/p4lY9IjHiWZETKmFq1mtU.jpeg
paper_authors.fullname: Yifan Zhang
paper_authors.type: user
paper_authors.user: yifAI
paper_authors._id: 66ff4115162623b07c7434e4
paper_authors.name: Ge Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-07T09:02:58.458Z
paper_authors._id: 638efcf4c67af472d316d424
paper_authors.avatarUrl: /avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
paper_authors.fullname: Ge Zhang
paper_authors.type: user
paper_authors.user: zhangysk
paper_authors._id: 66ff4115162623b07c7434e5
paper_authors.name: Yue Wu
paper_authors._id: 66ff4115162623b07c7434e6
paper_authors.name: Kangping Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-07T09:02:28.720Z
paper_authors._id: 664168ae50a41eda5160b960
paper_authors.avatarUrl: /avatars/bbb7fff94ebc9588b9d65e8ab5041633.svg
paper_authors.fullname: Kangping Xu
paper_authors.type: user
paper_authors.user: ScottXKP
paper_authors._id: 66ff4115162623b07c7434e7
paper_authors.name: Quanquan Gu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-07T09:02:22.173Z
paper_authors._id: 64c039128e2612254356bba5
paper_authors.avatarUrl: /avatars/06cc76feebba0cc80ebb8f4ff86f6d9b.svg
paper_authors.fullname: Quanquan Gu
paper_authors.type: user
paper_authors.user: thughost
paper_title: General Preference Modeling with Preference Representations for Aligning
    Language Models
paper_summary: Modeling human preferences is crucial for aligning foundation models with
  human values. Traditional reward modeling methods, such as the Bradley-Terry
  (BT) reward model, fall short in expressiveness, particularly in addressing
  intransitive preferences. Although supervised pair preference models (PairPM)
  can express general preferences, their implementation is highly ad-hoc and
  cannot guarantee a consistent preference probability of compared pairs.
  Additionally, they impose high computational costs due to their quadratic query
  complexity when comparing multiple responses. In this paper, we introduce
  preference representation learning, an approach that embeds responses into a
  latent space to capture intricate preference structures efficiently, achieving
  linear query complexity. Additionally, we propose preference score-based
  General Preference Optimization (GPO), which generalizes reward-based
  reinforcement learning from human feedback. Experimental results show that our
  General Preference representation model (GPM) outperforms the BT reward model
  on the RewardBench benchmark with a margin of up to 5.6% and effectively models
  cyclic preferences where any BT reward model behaves like a random guess.
  Furthermore, evaluations on downstream tasks such as AlpacaEval2.0 and
  MT-Bench, following the language model post-training with GPO and our general
  preference model, reveal substantial performance improvements with margins up
  to 9.3%. These findings indicate that our method may enhance the alignment of
  foundation models with nuanced human values. The code is available at
  https://github.com/general-preference/general-preference-model.
paper_ai_summary: Preference representation learning and General Preference Optimization (GPO) improve the modeling of human preferences, expressing complex and cyclic preferences more effectively than traditional methods.
paper_ai_keywords: Bradley-Terry (BT) reward model PairPM preference representation learning latent space preference score-based General Preference Optimization (GPO) reward-based reinforcement learning from human feedback RewardBench benchmark AlpacaEval2.0 MT-Bench