title: DPO-Shift: Shifting the Distribution of Direct Preference Optimization
summary: Direct Preference Optimization (DPO) and its variants have become
  increasingly popular for aligning language models with human preferences. These
  methods aim to teach models to better distinguish between chosen (or preferred)
  and rejected (or dispreferred) responses. However, prior research has
  identified that the probability of chosen responses often decreases during
  training, and this phenomenon is known as likelihood displacement. To tackle
  this challenge, in this work we introduce \method to controllably shift the
  distribution of the chosen probability. Then, we show that \method exhibits a
  fundamental trade-off between improving the chosen probability and sacrificing
  the reward margin, as supported by both theoretical analysis and experimental
  validation. Furthermore, we demonstrate the superiority of \method over DPO on
  downstream tasks such as MT-Bench and a designed win rate experiment. We
  believe this study shows that the likelihood displacement issue of DPO can be
  effectively mitigated with a simple, theoretically grounded solution. Our code
  is available at https://github.com/Meaquadddd/DPO-Shift.
paper_authors._id: 67ad5bd2ac32a8e230fc8996
paper_authors.name: Xiliang Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-18T16:39:57.092Z
paper_authors._id: 667fddfd10d2f8d7d62cb635
paper_authors.avatarUrl: /avatars/9c2a5cd059653fbbfe2ed0b73d50316d.svg
paper_authors.fullname: Xiliang Yang
paper_authors.type: user
paper_authors.user: NoManDeRY
paper_authors._id: 67ad5bd2ac32a8e230fc8997
paper_authors.name: Feng Jiang
paper_authors._id: 67ad5bd2ac32a8e230fc8998
paper_authors.name: Qianen Zhang
paper_authors._id: 67ad5bd2ac32a8e230fc8999
paper_authors.name: Lei Zhao
paper_authors._id: 67ad5bd2ac32a8e230fc899a
paper_authors.name: Xiao Li
paper_title: DPO-Shift: Shifting the Distribution of Direct Preference Optimization
paper_summary: Direct Preference Optimization (DPO) and its variants have become
  increasingly popular for aligning language models with human preferences. These
  methods aim to teach models to better distinguish between chosen (or preferred)
  and rejected (or dispreferred) responses. However, prior research has
  identified that the probability of chosen responses often decreases during
  training, and this phenomenon is known as likelihood displacement. To tackle
  this challenge, in this work we introduce \method to controllably shift the
  distribution of the chosen probability. Then, we show that \method exhibits a
  fundamental trade-off between improving the chosen probability and sacrificing
  the reward margin, as supported by both theoretical analysis and experimental
  validation. Furthermore, we demonstrate the superiority of \method over DPO on
  downstream tasks such as MT-Bench and a designed win rate experiment. We
  believe this study shows that the likelihood displacement issue of DPO can be
  effectively mitigated with a simple, theoretically grounded solution. Our code
  is available at https://github.com/Meaquadddd/DPO-Shift.
paper_ai_summary: A new method addresses likelihood displacement in Direct Preference Optimization by controlling the distribution of chosen responses, improving model performance on tasks like MT-Bench and win rate experiments.
paper_ai_keywords: Direct Preference Optimization DPO likelihood displacement chosen probability reward margin MT-Bench