Answers.Body: <p>I didn't watch the code but let me have a guess: For the reverse process, the last step from x(1) to x(0) should not add any noise respecting to f_sigma. Because this is the last step so adding any noise is not reasonable. So, just take x(1) as the final output is good enough to end the process.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-03T15:25:59.640Z
Answers.Text: Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer).
Answers.UserDisplayName: Community
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-06-29T17:10:47.633Z
Answers.LastActivityDate: 2023-06-29T17:10:47.633Z
Answers.UserDisplayName: 傅子恆
Body: <p>In the original diffusion model paper by Sohl-Dickstein et al., they explain very little about calculating the loss and training and network to learn the diffusion process. They did publish a repository with code <a href="https://github.com/Sohl-Dickstein/Diffusion-Probabilistic-Models/tree/master" rel="nofollow noreferrer">here</a>, which gives a few more clues.</p>
  <p>Now there is one thing I don't particularly understand, and that is that although the KL divergence is taken over <span class="math-container">$t=2..T$</span>,
  <a href="https://i.sstatic.net/KFUVt.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/KFUVt.png" alt="enter image description here" /></a></p>
  <p>in the code, they sample <span class="math-container">$t=1..T-1$</span> and say in the comments,</p>
  <pre><code># choose a timestep in [1, self.trajectory_length-1].
  # note the reverse process is fixed for the very
  # first timestep, so we skip it.
  </code></pre>
  <p>Now if I understand correctly, with the first timestep of the reverse process is just <span class="math-container">$t=T$</span>, which is just the isotropic gaussian, but what I don't understand is, why do they sample from <span class="math-container">$t=1$</span> instead of <span class="math-container">$t=2$</span> like in the KL divergence.</p>
  <p>Also, if you were indeed to sample from <span class="math-container">$t=2$</span>, how do you learn <span class="math-container">$f_{\mu}(x^1,1),f_{\Sigma}(x^1,1)$</span> so that you can reverse the last step? I would expect that to come form the entropy <span class="math-container">$H_q(X^{(1)}|X^{(0)})$</span>, but from the code we see that they replace that with the entropy of a Gaussian with <span class="math-container">$\sigma=\sqrt{1 - a_1} = \sqrt{b_1}$</span>,
  <span class="math-container">$$
  H_q(X^{(1)}|X^{(0)}) = \frac{1}{2}(\log 2 \pi + 1) + \frac{1}{2}\log b_1
  $$</span></p>
  <p>In the reverse process, you also see that they don't handle the last step of the reverse process any differently.</p>
  <p>Summarising, how can they sample <span class="math-container">$t \in 1..T-1$</span>, and calculate the KL divergence, while the equation specifies that <span class="math-container">$t \in 2..T$</span>?</p>
  <p><strong>EDIT</strong>: After analysing the code further, it looks like they are not learning the last denoising step, and are only generating <span class="math-container">$x^{(1)}$</span>.</p>
  
Tags: <papers><diffusion-models>
Title: In the original diffusion model paper, why do they sample the first step with the same loss?
UserDisplayName: Grieverheart
ThreadText: # In the original diffusion model paper, why do they sample the first step with the same loss?
  
  (Asked by: username_0 on 2023-06-23)
  
  In the original diffusion model paper by Sohl-Dickstein et al., they explain very little about calculating the loss and training and network to learn the diffusion process. They did publish a repository with code here, which gives a few more clues.
  Now there is one thing I don't particularly understand, and that is that although the KL divergence is taken over $t=2..T$,
  
  in the code, they sample $t=1..T-1$ and say in the comments,
  # choose a timestep in [1, self.trajectory_length-1].
  # note the reverse process is fixed for the very
  # first timestep, so we skip it.
  
  Now if I understand correctly, with the first timestep of the reverse process is just $t=T$, which is just the isotropic gaussian, but what I don't understand is, why do they sample from $t=1$ instead of $t=2$ like in the KL divergence.
  Also, if you were indeed to sample from $t=2$, how do you learn $f_{\mu}(x^1,1),f_{\Sigma}(x^1,1)$ so that you can reverse the last step? I would expect that to come form the entropy $H_q(X^{(1)}|X^{(0)})$, but from the code we see that they replace that with the entropy of a Gaussian with $\sigma=\sqrt{1 - a_1} = \sqrt{b_1}$,
  $$
  H_q(X^{(1)}|X^{(0)}) = \frac{1}{2}(\log 2 \pi + 1) + \frac{1}{2}\log b_1
  $$
  In the reverse process, you also see that they don't handle the last step of the reverse process any differently.
  Summarising, how can they sample $t \in 1..T-1$, and calculate the KL divergence, while the equation specifies that $t \in 2..T$?
  EDIT: After analysing the code further, it looks like they are not learning the last denoising step, and are only generating $x^{(1)}$.
  
  **Tags:** papers, diffusion-models
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-06-29. Score: 0
  
  I didn't watch the code but let me have a guess: For the reverse process, the last step from x(1) to x(0) should not add any noise respecting to f\_sigma. Because this is the last step so adding any noise is not reasonable. So, just take x(1) as the final output is good enough to end the process.
  
  > **Comments:**
  > **username_2**: Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer).  
  
  ---
  
  