Understanding Reference Policies in Direct Preference Optimization
Direct Preference Optimization (DPO) has become a widely used training method
for the instruction fine-tuning of large language models (LLMs). In this work,
we explore an under-investigated aspect of DPO - its dependency on the
reference model or policy. Such reference policies, typically instantiated as
the model to be further fine-tuned, are important since they can impose an
upper limit on DPO's effectiveness. Therefore, we address three related
research questions in this work. First, we explore the optimal strength of the
KL-divergence constraint in DPO, which penalizes deviations from the reference
policy, and find that DPO is sensitive to this strength. Next, we examine the
necessity of reference policies for instruction fine-tuning by providing both
theoretical and empirical comparisons between DPO and related learning
objectives, demonstrating DPO's superiority. Additionally, we investigate
whether DPO benefits from stronger reference policies, finding that a stronger
reference policy can lead to improved performance, but only when it is similar
to the model being fine-tuned. Our findings highlight the confounding role of
reference policies in DPO and offer insights for best practices, while also
identifying open research questions for future studies.
6699ed144ea6475a57eb5de6
Yixin Liu
extracted_confirmed
2024-07-19T06:46:56.221Z
6244de1c1c560fb11edfca44
/avatars/36558928bd04be7f49837d4c603681d7.svg
Yixin Liu
user
henryL7
6699ed144ea6475a57eb5de7
Pengfei Liu
claimed_verified
2025-06-29T10:15:19.312Z
6144a0c4ff1146bbd84d9865
https://cdn-avatars.huggingface.co/v1/production/uploads/1661715958139-6144a0c4ff1146bbd84d9865.png
Pengfei Liu
user
Pengfei
6699ed144ea6475a57eb5de8
Arman Cohan
Understanding Reference Policies in Direct Preference Optimization
Direct Preference Optimization (DPO) has become a widely used training method
for the instruction fine-tuning of large language models (LLMs). In this work,
we explore an under-investigated aspect of DPO - its dependency on the
reference model or policy. Such reference policies, typically instantiated as
the model to be further fine-tuned, are important since they can impose an
upper limit on DPO's effectiveness. Therefore, we address three related
research questions in this work. First, we explore the optimal strength of the
KL-divergence constraint in DPO, which penalizes deviations from the reference
policy, and find that DPO is sensitive to this strength. Next, we examine the
necessity of reference policies for instruction fine-tuning by providing both
theoretical and empirical comparisons between DPO and related learning
objectives, demonstrating DPO's superiority. Additionally, we investigate
whether DPO benefits from stronger reference policies, finding that a stronger
reference policy can lead to improved performance, but only when it is similar
to the model being fine-tuned. Our findings highlight the confounding role of
reference policies in DPO and offer insights for best practices, while also
identifying open research questions for future studies.
Direct Preference Optimization (DPO) is sensitive to KL-divergence constraints and benefits from reference policies similar to the model being fine-tuned, offering insights into best practices and future research.
KL-divergence Direct Preference Optimization (DPO) reference models reference policy instruction fine-tuning large language models (LLMs)