title: Self-Play Preference Optimization for Language Model Alignment
summary: Traditional reinforcement learning from human feedback (RLHF) approaches
  relying on parametric models like the Bradley-Terry model fall short in
  capturing the intransitivity and irrationality in human preferences. Recent
  advancements suggest that directly working with preference probabilities can
  yield a more accurate reflection of human preferences, enabling more flexible
  and accurate language model alignment. In this paper, we propose a
  self-play-based method for language model alignment, which treats the problem
  as a constant-sum two-player game aimed at identifying the Nash equilibrium
  policy. Our approach, dubbed Self-Play Preference Optimization (SPPO),
  approximates the Nash equilibrium through iterative policy updates and enjoys
  theoretical convergence guarantee. Our method can effectively increase the
  log-likelihood of the chosen response and decrease that of the rejected
  response, which cannot be trivially achieved by symmetric pairwise loss such as
  Direct Preference Optimization (DPO) and Identity Preference Optimization
  (IPO). In our experiments, using only 60k prompts (without responses) from the
  UltraFeedback dataset and without any prompt augmentation, by leveraging a
  pre-trained preference model PairRM with only 0.4B parameters, SPPO can obtain
  a model from fine-tuning Mistral-7B-Instruct-v0.2 that achieves the
  state-of-the-art length-controlled win-rate of 28.53% against GPT-4-Turbo on
  AlpacaEval 2.0. It also outperforms the (iterative) DPO and IPO on MT-Bench and
  the Open LLM Leaderboard. Notably, the strong performance of SPPO is achieved
  without additional external supervision (e.g., responses, preferences, etc.)
  from GPT-4 or other stronger language models.
paper_authors._id: 663307de888a4d3b08445692
paper_authors.name: Yue Wu
paper_authors._id: 663307de888a4d3b08445693
paper_authors.name: Zhiqing Sun
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-06T07:03:11.563Z
paper_authors._id: 5ee24411636bdb3834e2da24
paper_authors.avatarUrl: /avatars/0e12a0b19ada4138ddf808501ba68423.svg
paper_authors.fullname: Zhiqing Sun
paper_authors.type: user
paper_authors.user: zhiqings
paper_authors._id: 663307de888a4d3b08445694
paper_authors.name: Huizhuo Yuan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-03T07:16:36.552Z
paper_authors._id: 657be24e8d360b690d5b665f
paper_authors.avatarUrl: /avatars/423185c3f386e3fec2e5708c1030b013.svg
paper_authors.fullname: Angela Yuan
paper_authors.type: user
paper_authors.user: angelahzyuan
paper_authors._id: 663307de888a4d3b08445695
paper_authors.name: Kaixuan Ji
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-02T07:45:46.079Z
paper_authors._id: 656c1018e8bf55919a8234b0
paper_authors.avatarUrl: /avatars/0077cde47a96d91eb1d11b5a3809ead0.svg
paper_authors.fullname: Kaixuan Ji
paper_authors.type: user
paper_authors.user: KaixuanJi
paper_authors._id: 663307de888a4d3b08445696
paper_authors.name: Yiming Yang
paper_authors._id: 663307de888a4d3b08445697
paper_authors.name: Quanquan Gu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-02T07:09:22.583Z
paper_authors._id: 64c039128e2612254356bba5
paper_authors.avatarUrl: /avatars/06cc76feebba0cc80ebb8f4ff86f6d9b.svg
paper_authors.fullname: Quanquan Gu
paper_authors.type: user
paper_authors.user: thughost
paper_title: Self-Play Preference Optimization for Language Model Alignment
paper_summary: Traditional reinforcement learning from human feedback (RLHF) approaches
  relying on parametric models like the Bradley-Terry model fall short in
  capturing the intransitivity and irrationality in human preferences. Recent
  advancements suggest that directly working with preference probabilities can
  yield a more accurate reflection of human preferences, enabling more flexible
  and accurate language model alignment. In this paper, we propose a
  self-play-based method for language model alignment, which treats the problem
  as a constant-sum two-player game aimed at identifying the Nash equilibrium
  policy. Our approach, dubbed Self-Play Preference Optimization (SPPO),
  approximates the Nash equilibrium through iterative policy updates and enjoys
  theoretical convergence guarantee. Our method can effectively increase the
  log-likelihood of the chosen response and decrease that of the rejected
  response, which cannot be trivially achieved by symmetric pairwise loss such as
  Direct Preference Optimization (DPO) and Identity Preference Optimization
  (IPO). In our experiments, using only 60k prompts (without responses) from the
  UltraFeedback dataset and without any prompt augmentation, by leveraging a
  pre-trained preference model PairRM with only 0.4B parameters, SPPO can obtain
  a model from fine-tuning Mistral-7B-Instruct-v0.2 that achieves the
  state-of-the-art length-controlled win-rate of 28.53% against GPT-4-Turbo on
  AlpacaEval 2.0. It also outperforms the (iterative) DPO and IPO on MT-Bench and
  the Open LLM Leaderboard. Notably, the strong performance of SPPO is achieved
  without additional external supervision (e.g., responses, preferences, etc.)
  from GPT-4 or other stronger language models.
paper_ai_summary: A self-play method called SPPO for language model alignment achieves state-of-the-art performance by approximating Nash equilibrium policy in a constant-sum game setting, outperforming other approaches with limited data.
paper_ai_keywords: reinforcement learning from human feedback parametric models Bradley-Terry model preference probabilities self-play Nash equilibrium policy updates Direct Preference Optimization Identity Preference Optimization pre-trained preference model PairRM fine-tuning Mistral-7B-Instruct-v0.2 GPT-4-Turbo AlpacaEval 2.0 MT-Bench Open LLM Leaderboard