Answers.Body: <p>I had a problem concerning this and I think I can shed some light onto your question.</p>
  <p>We had a dataset of preferences generated for a really bad model, so naturally <span class="math-container">$y_w$</span> and <span class="math-container">$y_l$</span> were bad responses. If we were to use such preferences to train an already good SFT model, we would be looking into <span class="math-container">$\pi(y_w\vert x)$</span> or <span class="math-container">$\pi(y_l\vert x)$</span> for our good model, which is probably an extremely small number!</p>
  <p>In a sense we are telling it &quot;you should prefer to generate this bad answer rather than this atrocious one&quot;. This isn't helpful, and in my opinion will probably make the model worse.</p>
  <p>This can also happen if the model that the DPO data is coming from is radically different from the one we're fine tuning. If they produce really different text, <span class="math-container">$\pi(y_w\vert x)$</span> will again be extremely small, and we'd be comparing sentences that our network is highly unlikely to produce.</p>
  <p>Because of this I think it's highly desirable to evaluate responses coming from the same distribution, or at least a &quot;similar&quot; one. Hope this clarifies stuff for you.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-24T16:26:43.207Z
Answers.LastActivityDate: 2024-05-24T16:26:43.207Z
Answers.UserDisplayName: Guilherme Correa
Body: <p>I'm having trouble understanding this paragraph of the <a href="https://arxiv.org/pdf/2305.18290.pdf" rel="nofollow noreferrer">DPO paper</a>:</p>
  <p><a href="https://i.sstatic.net/gCdY4.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/gCdY4.png" alt="enter image description here" /></a></p>
  <p>Why does it matter so much that the preference data distribution aligns with the reference model output distribution? My understanding is that during training, the parameters of the sft (supervised fine-tuning) are updated such that chosen responses (<span class="math-container">$y_w$</span>) have a higher probability of being generated, and rejected responses (<span class="math-container">$y_l$</span>) have a lower probability of being generated, and the reference model is just there to prevent the sft model from straying too far from the original parameters. But I fail to understand how the wrong reference distribution could hinder this process. Could someone please help me?</p>
  
Tags: <natural-language-processing><large-language-models><rlhf>
Title: understanding the distribution shift problem in direct preference optimization (DPO)
UserDisplayName: Ivy Cao
ThreadText: # understanding the distribution shift problem in direct preference optimization (DPO)
  
  (Asked by: username_0 on 2024-01-29)
  
  I'm having trouble understanding this paragraph of the [DPO paper](https://arxiv.org/pdf/2305.18290.pdf):
  
  [![enter image description here](https://i.sstatic.net/gCdY4.png)](https://i.sstatic.net/gCdY4.png)
  
  Why does it matter so much that the preference data distribution aligns with the reference model output distribution? My understanding is that during training, the parameters of the sft (supervised fine-tuning) are updated such that chosen responses ($y\_w$) have a higher probability of being generated, and rejected responses ($y\_l$) have a lower probability of being generated, and the reference model is just there to prevent the sft model from straying too far from the original parameters. But I fail to understand how the wrong reference distribution could hinder this process. Could someone please help me?
  
  **Tags:** natural-language-processing, large-language-models, rlhf
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-05-24. Score: 0
  
  I had a problem concerning this and I think I can shed some light onto your question.
  
  We had a dataset of preferences generated for a really bad model, so naturally $y\_w$ and $y\_l$ were bad responses. If we were to use such preferences to train an already good SFT model, we would be looking into $\pi(y\_w\vert x)$ or $\pi(y\_l\vert x)$ for our good model, which is probably an extremely small number!
  
  In a sense we are telling it "you should prefer to generate this bad answer rather than this atrocious one". This isn't helpful, and in my opinion will probably make the model worse.
  
  This can also happen if the model that the DPO data is coming from is radically different from the one we're fine tuning. If they produce really different text, $\pi(y\_w\vert x)$ will again be extremely small, and we'd be comparing sentences that our network is highly unlikely to produce.
  
  Because of this I think it's highly desirable to evaluate responses coming from the same distribution, or at least a "similar" one. Hope this clarifies stuff for you.
  
  
  ---
  
  