General Preference Modeling with Preference Representations for Aligning
  Language Models
Modeling human preferences is crucial for aligning foundation models with
human values. Traditional reward modeling methods, such as the Bradley-Terry
(BT) reward model, fall short in expressiveness, particularly in addressing
intransitive preferences. Although supervised pair preference models (PairPM)
can express general preferences, their implementation is highly ad-hoc and
cannot guarantee a consistent preference probability of compared pairs.
Additionally, they impose high computational costs due to their quadratic query
complexity when comparing multiple responses. In this paper, we introduce
preference representation learning, an approach that embeds responses into a
latent space to capture intricate preference structures efficiently, achieving
linear query complexity. Additionally, we propose preference score-based
General Preference Optimization (GPO), which generalizes reward-based
reinforcement learning from human feedback. Experimental results show that our
General Preference representation model (GPM) outperforms the BT reward model
on the RewardBench benchmark with a margin of up to 5.6% and effectively models
cyclic preferences where any BT reward model behaves like a random guess.
Furthermore, evaluations on downstream tasks such as AlpacaEval2.0 and
MT-Bench, following the language model post-training with GPO and our general
preference model, reveal substantial performance improvements with margins up
to 9.3%. These findings indicate that our method may enhance the alignment of
foundation models with nuanced human values. The code is available at
https://github.com/general-preference/general-preference-model.
66ff4115162623b07c7434e3
Yifan Zhang
claimed_verified
2024-10-04T07:31:15.090Z
647bf082aba7062fe5c51ca9
https://cdn-avatars.huggingface.co/v1/production/uploads/647bf082aba7062fe5c51ca9/p4lY9IjHiWZETKmFq1mtU.jpeg
Yifan Zhang
user
yifAI
66ff4115162623b07c7434e4
Ge Zhang
admin_assigned
2024-10-07T09:02:58.458Z
638efcf4c67af472d316d424
/avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
Ge Zhang
user
zhangysk
66ff4115162623b07c7434e5
Yue Wu
66ff4115162623b07c7434e6
Kangping Xu
admin_assigned
2024-10-07T09:02:28.720Z
664168ae50a41eda5160b960
/avatars/bbb7fff94ebc9588b9d65e8ab5041633.svg
Kangping Xu
user
ScottXKP
66ff4115162623b07c7434e7
Quanquan Gu
admin_assigned
2024-10-07T09:02:22.173Z
64c039128e2612254356bba5
/avatars/06cc76feebba0cc80ebb8f4ff86f6d9b.svg
Quanquan Gu
user
thughost
General Preference Modeling with Preference Representations for Aligning
  Language Models
Modeling human preferences is crucial for aligning foundation models with
human values. Traditional reward modeling methods, such as the Bradley-Terry
(BT) reward model, fall short in expressiveness, particularly in addressing
intransitive preferences. Although supervised pair preference models (PairPM)
can express general preferences, their implementation is highly ad-hoc and
cannot guarantee a consistent preference probability of compared pairs.
Additionally, they impose high computational costs due to their quadratic query
complexity when comparing multiple responses. In this paper, we introduce
preference representation learning, an approach that embeds responses into a
latent space to capture intricate preference structures efficiently, achieving
linear query complexity. Additionally, we propose preference score-based
General Preference Optimization (GPO), which generalizes reward-based
reinforcement learning from human feedback. Experimental results show that our
General Preference representation model (GPM) outperforms the BT reward model
on the RewardBench benchmark with a margin of up to 5.6% and effectively models
cyclic preferences where any BT reward model behaves like a random guess.
Furthermore, evaluations on downstream tasks such as AlpacaEval2.0 and
MT-Bench, following the language model post-training with GPO and our general
preference model, reveal substantial performance improvements with margins up
to 9.3%. These findings indicate that our method may enhance the alignment of
foundation models with nuanced human values. The code is available at
https://github.com/general-preference/general-preference-model.
Preference representation learning and General Preference Optimization (GPO) improve the modeling of human preferences, expressing complex and cyclic preferences more effectively than traditional methods.
Bradley-Terry (BT) reward model PairPM preference representation learning latent space preference score-based General Preference Optimization (GPO) reward-based reinforcement learning from human feedback RewardBench benchmark AlpacaEval2.0 MT-Bench