Self-Play Preference Optimization for Language Model Alignment
Traditional reinforcement learning from human feedback (RLHF) approaches
relying on parametric models like the Bradley-Terry model fall short in
capturing the intransitivity and irrationality in human preferences. Recent
advancements suggest that directly working with preference probabilities can
yield a more accurate reflection of human preferences, enabling more flexible
and accurate language model alignment. In this paper, we propose a
self-play-based method for language model alignment, which treats the problem
as a constant-sum two-player game aimed at identifying the Nash equilibrium
policy. Our approach, dubbed Self-Play Preference Optimization (SPPO),
approximates the Nash equilibrium through iterative policy updates and enjoys
theoretical convergence guarantee. Our method can effectively increase the
log-likelihood of the chosen response and decrease that of the rejected
response, which cannot be trivially achieved by symmetric pairwise loss such as
Direct Preference Optimization (DPO) and Identity Preference Optimization
(IPO). In our experiments, using only 60k prompts (without responses) from the
UltraFeedback dataset and without any prompt augmentation, by leveraging a
pre-trained preference model PairRM with only 0.4B parameters, SPPO can obtain
a model from fine-tuning Mistral-7B-Instruct-v0.2 that achieves the
state-of-the-art length-controlled win-rate of 28.53% against GPT-4-Turbo on
AlpacaEval 2.0. It also outperforms the (iterative) DPO and IPO on MT-Bench and
the Open LLM Leaderboard. Notably, the strong performance of SPPO is achieved
without additional external supervision (e.g., responses, preferences, etc.)
from GPT-4 or other stronger language models.
663307de888a4d3b08445692
Yue Wu
663307de888a4d3b08445693
Zhiqing Sun
claimed_verified
2024-05-06T07:03:11.563Z
5ee24411636bdb3834e2da24
/avatars/0e12a0b19ada4138ddf808501ba68423.svg
Zhiqing Sun
user
zhiqings
663307de888a4d3b08445694
Huizhuo Yuan
claimed_verified
2024-05-03T07:16:36.552Z
657be24e8d360b690d5b665f
/avatars/423185c3f386e3fec2e5708c1030b013.svg
Angela Yuan
user
angelahzyuan
663307de888a4d3b08445695
Kaixuan Ji
admin_assigned
2024-05-02T07:45:46.079Z
656c1018e8bf55919a8234b0
/avatars/0077cde47a96d91eb1d11b5a3809ead0.svg
Kaixuan Ji
user
KaixuanJi
663307de888a4d3b08445696
Yiming Yang
663307de888a4d3b08445697
Quanquan Gu
claimed_verified
2024-05-02T07:09:22.583Z
64c039128e2612254356bba5
/avatars/06cc76feebba0cc80ebb8f4ff86f6d9b.svg
Quanquan Gu
user
thughost
Self-Play Preference Optimization for Language Model Alignment
Traditional reinforcement learning from human feedback (RLHF) approaches
relying on parametric models like the Bradley-Terry model fall short in
capturing the intransitivity and irrationality in human preferences. Recent
advancements suggest that directly working with preference probabilities can
yield a more accurate reflection of human preferences, enabling more flexible
and accurate language model alignment. In this paper, we propose a
self-play-based method for language model alignment, which treats the problem
as a constant-sum two-player game aimed at identifying the Nash equilibrium
policy. Our approach, dubbed Self-Play Preference Optimization (SPPO),
approximates the Nash equilibrium through iterative policy updates and enjoys
theoretical convergence guarantee. Our method can effectively increase the
log-likelihood of the chosen response and decrease that of the rejected
response, which cannot be trivially achieved by symmetric pairwise loss such as
Direct Preference Optimization (DPO) and Identity Preference Optimization
(IPO). In our experiments, using only 60k prompts (without responses) from the
UltraFeedback dataset and without any prompt augmentation, by leveraging a
pre-trained preference model PairRM with only 0.4B parameters, SPPO can obtain
a model from fine-tuning Mistral-7B-Instruct-v0.2 that achieves the
state-of-the-art length-controlled win-rate of 28.53% against GPT-4-Turbo on
AlpacaEval 2.0. It also outperforms the (iterative) DPO and IPO on MT-Bench and
the Open LLM Leaderboard. Notably, the strong performance of SPPO is achieved
without additional external supervision (e.g., responses, preferences, etc.)
from GPT-4 or other stronger language models.
A self-play method called SPPO for language model alignment achieves state-of-the-art performance by approximating Nash equilibrium policy in a constant-sum game setting, outperforming other approaches with limited data.
reinforcement learning from human feedback parametric models Bradley-Terry model preference probabilities self-play Nash equilibrium policy updates Direct Preference Optimization Identity Preference Optimization pre-trained preference model PairRM fine-tuning Mistral-7B-Instruct-v0.2 GPT-4-Turbo AlpacaEval 2.0 MT-Bench Open LLM Leaderboard