Body: <p>There is a <a href="https://github.com/huggingface/trl/blob/d708ec272f292ded00a4d0f14b5bd214741b87f3/trl/trainer/dpo_trainer.py#L762" rel="nofollow noreferrer">reference_free</a> parameter in <a href="https://github.com/huggingface/trl" rel="nofollow noreferrer">trl</a>'s loss function implementation of DPO, while <a href="https://arxiv.org/abs/2305.18290" rel="nofollow noreferrer">the original DPO paper</a> does not mention the concept of &quot;reference free&quot;.</p>
  <p>In trl's implementation:</p>
  <pre><code>    pi_logratios = policy_chosen_logps - policy_rejected_logps
      if reference_free:
          ref_logratios = 0
      else:
          ref_logratios = reference_chosen_logps - reference_rejected_logps
  
      logits = pi_logratios - ref_logratios
  </code></pre>
  <p>When <code>reference_free</code> is <code>True</code>, the loss is calculated as cross entropy between <code>policy_chosen_logps</code> and <code>policy_rejected_logps</code>, while <code>reference_chosen_logps</code> and <code>reference_rejected_logps</code> are not in use.</p>
  <p>What are the pros and cons of the &quot;reference free&quot; DPO?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-12-22T22:17:15.927Z
Comments.Text: Hi @Yang Bo and welcome to AI Stack Exchange! To receive an answer on this question, it might help to add a few more details. Thank you for posting, and we hope to see more of your questions on this site in the near future!
Comments.UserDisplayName: DeepQZero
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-12-23T00:05:21.680Z
Comments.Text: @DeepQZero Thank you for your warm welcome message. I just added the related code and my interpretation of the implementation.
Comments.UserDisplayName: Yang Bo
Tags: <neural-networks><reinforcement-learning><natural-language-processing><large-language-models><loss>
Title: Why is there a "reference free" option in DPO (Direct Preference Optimization)'s loss function?
UserDisplayName: Yang Bo
ThreadText: # Why is there a "reference free" option in DPO (Direct Preference Optimization)'s loss function?
  
  (Asked by: username_0 on 2023-12-16)
  
  There is a reference_free parameter in trl's loss function implementation of DPO, while the original DPO paper does not mention the concept of "reference free".
  In trl's implementation:
      pi_logratios = policy_chosen_logps - policy_rejected_logps
      if reference_free:
          ref_logratios = 0
      else:
          ref_logratios = reference_chosen_logps - reference_rejected_logps
  
      logits = pi_logratios - ref_logratios
  
  When reference_free is True, the loss is calculated as cross entropy between policy_chosen_logps and policy_rejected_logps, while reference_chosen_logps and reference_rejected_logps are not in use.
  What are the pros and cons of the "reference free" DPO?
  
  **Tags:** neural-networks, reinforcement-learning, natural-language-processing, large-language-models, loss
  > **Comments:**
  > **username_1**: Hi @Yang Bo and welcome to AI Stack Exchange! To receive an answer on this question, it might help to add a few more details. Thank you for posting, and we hope to see more of your questions on this site in the near future!  
  > **username_0**: @DeepQZero Thank you for your warm welcome message. I just added the related code and my interpretation of the implementation.  
  
  ---
  
  ## 0 answers
  
  