DPO-Shift: Shifting the Distribution of Direct Preference Optimization
Direct Preference Optimization (DPO) and its variants have become
increasingly popular for aligning language models with human preferences. These
methods aim to teach models to better distinguish between chosen (or preferred)
and rejected (or dispreferred) responses. However, prior research has
identified that the probability of chosen responses often decreases during
training, and this phenomenon is known as likelihood displacement. To tackle
this challenge, in this work we introduce \method to controllably shift the
distribution of the chosen probability. Then, we show that \method exhibits a
fundamental trade-off between improving the chosen probability and sacrificing
the reward margin, as supported by both theoretical analysis and experimental
validation. Furthermore, we demonstrate the superiority of \method over DPO on
downstream tasks such as MT-Bench and a designed win rate experiment. We
believe this study shows that the likelihood displacement issue of DPO can be
effectively mitigated with a simple, theoretically grounded solution. Our code
is available at https://github.com/Meaquadddd/DPO-Shift.
67ad5bd2ac32a8e230fc8996
Xiliang Yang
claimed_verified
2025-02-18T16:39:57.092Z
667fddfd10d2f8d7d62cb635
/avatars/9c2a5cd059653fbbfe2ed0b73d50316d.svg
Xiliang Yang
user
NoManDeRY
67ad5bd2ac32a8e230fc8997
Feng Jiang
67ad5bd2ac32a8e230fc8998
Qianen Zhang
67ad5bd2ac32a8e230fc8999
Lei Zhao
67ad5bd2ac32a8e230fc899a
Xiao Li
DPO-Shift: Shifting the Distribution of Direct Preference Optimization
Direct Preference Optimization (DPO) and its variants have become
increasingly popular for aligning language models with human preferences. These
methods aim to teach models to better distinguish between chosen (or preferred)
and rejected (or dispreferred) responses. However, prior research has
identified that the probability of chosen responses often decreases during
training, and this phenomenon is known as likelihood displacement. To tackle
this challenge, in this work we introduce \method to controllably shift the
distribution of the chosen probability. Then, we show that \method exhibits a
fundamental trade-off between improving the chosen probability and sacrificing
the reward margin, as supported by both theoretical analysis and experimental
validation. Furthermore, we demonstrate the superiority of \method over DPO on
downstream tasks such as MT-Bench and a designed win rate experiment. We
believe this study shows that the likelihood displacement issue of DPO can be
effectively mitigated with a simple, theoretically grounded solution. Our code
is available at https://github.com/Meaquadddd/DPO-Shift.
A new method addresses likelihood displacement in Direct Preference Optimization by controlling the distribution of chosen responses, improving model performance on tasks like MT-Bench and win rate experiments.
Direct Preference Optimization DPO likelihood displacement chosen probability reward margin MT-Bench