Body: <p><a href="https://i.sstatic.net/657AD55B.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/657AD55B.png" alt="enter image description here" /></a></p>
  <p>The above image captures an excerpt from the original DPO paper.</p>
  <p><strong>My current understanding of the DPO process is as follows:</strong></p>
  <ol>
  <li>First, initialize both the policy model and the reference model from the SFT model.</li>
  <li>For each prompt, the reference model generates a pair of answers, which are then labeled by human annotators to create a human preference dataset in an offline manner.</li>
  <li>Minimize the DPO loss to continuously optimize the policy model.</li>
  </ol>
  <p>However, there's one point in this passage that I don't quite understand. It mentions that, in practice, people often prefer to use publicly available preference datasets rather than generating their own privately. Since the preference dataset is obtained through sampling from the SFT model, we initialize the reference model from the SFT model when it is available. If the SFT model is unavailable, we need to initialize a reference model ourselves according to the corresponding objective function.</p>
  <p><strong>The parts I don't understand are as follows:</strong></p>
  <ol>
  <li>The policy model is also initialized from the SFT model. If the SFT model is unavailable, where does the policy model come from?</li>
  <li>When the SFT model is unavailable, the reference model is trained using the objective function provided in the paper. What is its initial state then? Is it randomly initialized and then trained according to the preference responses?</li>
  <li>Is my understanding of the DPO process correct?</li>
  </ol>
  <p>Thank you in advance for your insights and assistance!</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-09-02T08:56:11.083Z
Comments.Text: Please, one specific question per post.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-09-02T15:47:12.793Z
Comments.Text: @nbro As the title suggests, I did only ask one specific question.
Comments.UserDisplayName: korangar leo
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-09-08T12:31:19.693Z
Comments.Text: Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.
Comments.UserDisplayName: Community
Tags: <reinforcement-learning><natural-language-processing><large-language-models>
Title: How should the DPO algorithm be executed when the SFT model is unavailable?
UserDisplayName: korangar leo
ThreadText: # How should the DPO algorithm be executed when the SFT model is unavailable?
  
  (Asked by: username_0 on 2024-08-28)
  
  [![enter image description here](https://i.sstatic.net/657AD55B.png)](https://i.sstatic.net/657AD55B.png)
  
  The above image captures an excerpt from the original DPO paper.
  
  **My current understanding of the DPO process is as follows:**
  
  1. First, initialize both the policy model and the reference model from the SFT model.
  2. For each prompt, the reference model generates a pair of answers, which are then labeled by human annotators to create a human preference dataset in an offline manner.
  3. Minimize the DPO loss to continuously optimize the policy model.
  
  However, there's one point in this passage that I don't quite understand. It mentions that, in practice, people often prefer to use publicly available preference datasets rather than generating their own privately. Since the preference dataset is obtained through sampling from the SFT model, we initialize the reference model from the SFT model when it is available. If the SFT model is unavailable, we need to initialize a reference model ourselves according to the corresponding objective function.
  
  **The parts I don't understand are as follows:**
  
  1. The policy model is also initialized from the SFT model. If the SFT model is unavailable, where does the policy model come from?
  2. When the SFT model is unavailable, the reference model is trained using the objective function provided in the paper. What is its initial state then? Is it randomly initialized and then trained according to the preference responses?
  3. Is my understanding of the DPO process correct?
  
  Thank you in advance for your insights and assistance!
  
  **Tags:** reinforcement-learning, natural-language-processing, large-language-models
  
  > **Comments:**
  > **username_1**: Please, one specific question per post.  
  > **username_0**: @username_1 As the title suggests, I did only ask one specific question.  
  > **username_2**: Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.  
  
  ---
  
  ## 0 answers
  
  