<p>There is a <a href="https://github.com/huggingface/trl/blob/d708ec272f292ded00a4d0f14b5bd214741b87f3/trl/trainer/dpo_trainer.py#L762" rel="nofollow noreferrer">reference_free</a> parameter in <a href="https://github.com/huggingface/trl" rel="nofollow noreferrer">trl</a>'s loss function implementation of DPO, while <a href="https://arxiv.org/abs/2305.18290" rel="nofollow noreferrer">the original DPO paper</a> does not mention the concept of &quot;reference free&quot;.</p>
<p>In trl's implementation:</p>
<pre><code>    pi_logratios = policy_chosen_logps - policy_rejected_logps
    if reference_free:
        ref_logratios = 0
    else:
        ref_logratios = reference_chosen_logps - reference_rejected_logps

    logits = pi_logratios - ref_logratios
</code></pre>
<p>When <code>reference_free</code> is <code>True</code>, the loss is calculated as cross entropy between <code>policy_chosen_logps</code> and <code>policy_rejected_logps</code>, while <code>reference_chosen_logps</code> and <code>reference_rejected_logps</code> are not in use.</p>
<p>What are the pros and cons of the &quot;reference free&quot; DPO?</p>

CC BY-SA 4.0
2023-12-22T22:17:15.927Z
Hi @Yang Bo and welcome to AI Stack Exchange! To receive an answer on this question, it might help to add a few more details. Thank you for posting, and we hope to see more of your questions on this site in the near future!
DeepQZero
CC BY-SA 4.0
2023-12-23T00:05:21.680Z
@DeepQZero Thank you for your warm welcome message. I just added the related code and my interpretation of the implementation.
Yang Bo
<neural-networks><reinforcement-learning><natural-language-processing><large-language-models><loss>
Why is there a "reference free" option in DPO (Direct Preference Optimization)'s loss function?
Yang Bo
# Why is there a "reference free" option in DPO (Direct Preference Optimization)'s loss function?

(Asked by: username_0 on 2023-12-16)

There is a reference_free parameter in trl's loss function implementation of DPO, while the original DPO paper does not mention the concept of "reference free".
In trl's implementation:
    pi_logratios = policy_chosen_logps - policy_rejected_logps
    if reference_free:
        ref_logratios = 0
    else:
        ref_logratios = reference_chosen_logps - reference_rejected_logps

    logits = pi_logratios - ref_logratios

When reference_free is True, the loss is calculated as cross entropy between policy_chosen_logps and policy_rejected_logps, while reference_chosen_logps and reference_rejected_logps are not in use.
What are the pros and cons of the "reference free" DPO?

**Tags:** neural-networks, reinforcement-learning, natural-language-processing, large-language-models, loss

> **Comments:**
> **username_1**: Hi @username_0 and welcome to AI Stack Exchange! To receive an answer on this question, it might help to add a few more details. Thank you for posting, and we hope to see more of your questions on this site in the near future!  
> **username_0**: @username_1 Thank you for your warm welcome message. I just added the related code and my interpretation of the implementation.  

---

## 0 answers

