title: Understanding Reference Policies in Direct Preference Optimization
summary: Direct Preference Optimization (DPO) has become a widely used training method
  for the instruction fine-tuning of large language models (LLMs). In this work,
  we explore an under-investigated aspect of DPO - its dependency on the
  reference model or policy. Such reference policies, typically instantiated as
  the model to be further fine-tuned, are important since they can impose an
  upper limit on DPO's effectiveness. Therefore, we address three related
  research questions in this work. First, we explore the optimal strength of the
  KL-divergence constraint in DPO, which penalizes deviations from the reference
  policy, and find that DPO is sensitive to this strength. Next, we examine the
  necessity of reference policies for instruction fine-tuning by providing both
  theoretical and empirical comparisons between DPO and related learning
  objectives, demonstrating DPO's superiority. Additionally, we investigate
  whether DPO benefits from stronger reference policies, finding that a stronger
  reference policy can lead to improved performance, but only when it is similar
  to the model being fine-tuned. Our findings highlight the confounding role of
  reference policies in DPO and offer insights for best practices, while also
  identifying open research questions for future studies.
paper_authors._id: 6699ed144ea6475a57eb5de6
paper_authors.name: Yixin Liu
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-07-19T06:46:56.221Z
paper_authors._id: 6244de1c1c560fb11edfca44
paper_authors.avatarUrl: /avatars/36558928bd04be7f49837d4c603681d7.svg
paper_authors.fullname: Yixin Liu
paper_authors.type: user
paper_authors.user: henryL7
paper_authors._id: 6699ed144ea6475a57eb5de7
paper_authors.name: Pengfei Liu
paper_authors._id: 6699ed144ea6475a57eb5de8
paper_authors.name: Arman Cohan
paper_title: Understanding Reference Policies in Direct Preference Optimization
paper_summary: Direct Preference Optimization (DPO) has become a widely used training method
  for the instruction fine-tuning of large language models (LLMs). In this work,
  we explore an under-investigated aspect of DPO - its dependency on the
  reference model or policy. Such reference policies, typically instantiated as
  the model to be further fine-tuned, are important since they can impose an
  upper limit on DPO's effectiveness. Therefore, we address three related
  research questions in this work. First, we explore the optimal strength of the
  KL-divergence constraint in DPO, which penalizes deviations from the reference
  policy, and find that DPO is sensitive to this strength. Next, we examine the
  necessity of reference policies for instruction fine-tuning by providing both
  theoretical and empirical comparisons between DPO and related learning
  objectives, demonstrating DPO's superiority. Additionally, we investigate
  whether DPO benefits from stronger reference policies, finding that a stronger
  reference policy can lead to improved performance, but only when it is similar
  to the model being fine-tuned. Our findings highlight the confounding role of
  reference policies in DPO and offer insights for best practices, while also
  identifying open research questions for future studies.
paper_ai_keywords: Direct Preference Optimization (DPO) instruction fine-tuning large language models (LLMs) reference model policy KL-divergence constraint reference policies theoretical comparisons empirical comparisons learning objectives