Answers.Body: <p>I am also learning diffusion models and would like to give some information.</p>
  <blockquote>
  <p>At this point, I don't understand the role of the model <span class="math-container">$p_\theta$</span></p>
  </blockquote>
  <p>To clear a bit: <span class="math-container">$p_\theta$</span> is just another annotation for U-net and the role is receiving (<span class="math-container">$x_t$</span>,<span class="math-container">$t$</span>) (sometimes also receives classifier <span class="math-container">$y$</span>) and predicts <span class="math-container">$x_0$</span> OR <span class="math-container">$x_{t-1}$</span> depending on different papers. So at the end of the day, to synthesize new data, given a noisy (usually Gaussian) image, U-net can iteratively predict <span class="math-container">$x_0$</span> better - check out algorithm 2 in the DDPM paper (2020).</p>
  <p>Your question about the posterior might be answered in more detail here: <a href="https://www.youtube.com/watch?v=HoKDTa5jHvg" rel="nofollow noreferrer">Diffusion Models | Paper Explanation | Math Explained - YouTube</a></p>
  <p>Check video time around <strong>18:00</strong> that explains a bit more information regarding <span class="math-container">$x_0$</span> guided process in the optimization of lower boundary.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-23T10:36:57.713Z
Answers.Text: Still can't find an answer to the question: Why in the DDIM a noise is added every step of the reverse process, and if it makes it better.
Answers.UserDisplayName: Nathan G
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-24T09:15:32.850Z
Answers.Text: Why is noise added at each timestamp during the reverse process?
Answers.UserDisplayName: Nathan G
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-25T17:34:05.723Z
Answers.Text: @NathanB I am assuming you mean the DDIM diffusion way. DDIM is predicting x_0 and adding the noise back to x_t-1 during reverse. From my understanding, this strategy is turning the diffusion process from markovian to non-markovian. The benefits of DDIM include faster generation, better sampling strategy, etc. I would recommend the cold diffusion paper and the original DDIM paper for a deeper understanding.
Answers.UserDisplayName: lqi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-26T11:32:05.793Z
Answers.Text: What will happen if you don't add some noise in the reverse process at every step? After all, you start from a random noise image, so you already have the randomness needed for the sampling, why add more noise every step backwards?
Answers.UserDisplayName: Nathan G
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-27T18:48:53.903Z
Answers.Text: @NathanB you can do it without adding noise during sampling. It's just different sampling strategies. In the original DDPM paper, they are not adding noise when sampling backward.
Answers.UserDisplayName: lqi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-29T11:22:33.993Z
Answers.Text: Is the purpose to basically add more 'randomness' to the process so that the result will be more 'creative'?
Answers.UserDisplayName: Nathan G
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-30T02:20:45.000Z
Answers.Text: @NathanB I can't give you the conclusion that it's adding more "randomness" to be more "creative" as I haven't fully researched it enough. However, the main purpose includes changing non-markovian to markovian, which allows faster generation. Different sampling strategies have different pros and cons, I highly suggest reading papers like cold diffusion, stable diffusion, ILVR, etc. to get a deeper intuition of those sampling strategies.
Answers.UserDisplayName: lqi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-09-25T20:29:00.547Z
Answers.LastActivityDate: 2023-01-25T17:40:19.727Z
Answers.LastEditDate: 2023-01-25T17:40:19.727Z
Answers.UserDisplayName: lqi
Answers.Body: <p>You do not yet have <span class="math-container">$\mathbf{x}_0$</span> during sampling (not training).  That's why you need to approximate <span class="math-container">$q(\mathbf{x}_{t−1}|\mathbf{x}_t, \mathbf{x}_0)$</span> with <span class="math-container">$p_{\theta}(\mathbf{x}_{t−1}|\mathbf{x}_t)$</span> via variational inference such as KL divergence.  After training with good data, this should produce an approximation of <span class="math-container">$\mathbf{x}_0$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-19T19:59:59.033Z
Answers.LastActivityDate: 2022-11-23T20:24:59.473Z
Answers.LastEditDate: 2022-11-23T20:24:59.473Z
Answers.UserDisplayName: lkjsfkshd
Body: <p>I'm reading this interesting <a href="https://lilianweng.github.io/posts/2021-07-11-diffusion-models/" rel="noreferrer">blog post</a> explaining <em>diffusion probabilistic models</em> and trying to understand the following.</p>
  <p>In order to compute the <em>reverse process</em>, we need to consider the posterior distribution <span class="math-container">$q(\textbf{x}_{t-1} | \textbf{x}_t)$</span> which is said to be <strong>intractable</strong>*</p>
  <blockquote>
  <p>because <strong>it needs to use the entire dataset</strong> and therefore we need to learn a model <span class="math-container">$p_\theta$</span> to approximate these conditional probabilities in order to run the reverse diffusion process.</p>
  </blockquote>
  <p>If we use <em>Bayes theorem</em> we have</p>
  <p><span class="math-container">$$q(\textbf{x}_{t-1} | \textbf{x}_t) = \frac{q(\textbf{x}_t |\textbf{x}_{t-1})q(\textbf{x}_{t-1})}{q(\textbf{x}_t)}$$</span></p>
  <p>I understand that indeed we don't have any prior knowledge of <span class="math-container">$q(\textbf{x}_{t-1})$</span> or <span class="math-container">$q(\textbf{x}_t)$</span> since this would mean already having the distribution we are trying to estimate. Is this correct?</p>
  <p>The above posterior becomes <strong>tractable</strong> when conditioned on <span class="math-container">$\textbf{x}_0$</span> and we obtain</p>
  <p><span class="math-container">$$q(\textbf{x}_{t-1} | \textbf{x}_t , \textbf{x}_0) = \mathcal{N}(\tilde{\bf{\mu}}(\textbf{x}_t , \textbf{x}_0) \, , \, \tilde{\beta}_t \textbf{I})$$</span></p>
  <p>So, apparently, we obtain a <strong>posterior</strong> that can be calculated in <strong>closed form</strong> when we condition on the original data <span class="math-container">$\textbf{x}_0$</span>. At this point, I don't understand the role of the model <span class="math-container">$p_\theta$</span> : why do we need to tune the parameters of a model if we can already obtain our posterior?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-03T10:53:48.340Z
Comments.Text: Don't know much about diffusion models, but I think this iterative conversion of $\mathbf{x}_0$ to a gaussian is only being done to then train a model to reverse the process. So in my understanding, the whole idea is to not use $\mathbf{x}_0$, otherwise, why would you want to add noise to $\mathbf{x}_0$ in the first place?
  This might be comparable to the encoding and decoding process of variational autoencoders. The encoding makes the input noisier and the decoder reconstructs the input. Here, you only condition the encoder on $\mathbf{x}_0$, but not the decoder directly.
Comments.UserDisplayName: Chillston
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-03T12:10:13.947Z
Comments.Text: @Chillston Thank you greatly for your answer! So when we are optimizing the lower bound on the negative log-likelihood even there we are not using $\mathbf{x}_0$  correct?  Since I have you here maybe you can also tell me the intuition behind setting up a variational inference problem with a fixed approximate posterior, I thought the whole point of variational inference was to allow the posterior to belong to a big enough (well-behaved) class so as to get as close as possible to the true posterior in an efficient way.
Comments.UserDisplayName: Monolite
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-03T15:36:55.650Z
Comments.Text: I hope I understand you right, I really didn't look into diffusion models too deep, yet. How I understand this is: You ultimately want a model that denoises a given random sample and maps it onto the manifold of your data. Because it is not feasible to model $p_{\Theta} : X_T \rightarrow X_0$ directly, the model learns a single denoising step $p_{\Theta} : X_{t} \rightarrow X_{t+1}$ ($0 \leq t \leq T$). By iteratively applying the model you arrive at $X_0$ after $T$ iterations. Thus given a noisy sample $x_t$, the model predicts the noise that was added to $\mathbf{x}_{t-1}$.
Comments.UserDisplayName: Chillston
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-03T15:37:10.263Z
Comments.Text: So the (simplified) training objective is actually the MSE between predicted and actual noise (s. Equation 14 [Ho et al.](https://arxiv.org/pdf/2006.11239.pdf)). So I'd say you need $\mathbf{x}_0$ to optimize the variational bound.
Comments.UserDisplayName: Chillston
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-03T15:38:18.673Z
Comments.Text: Regarding the second part: In my intuition, having a fixed posterior doesn't limit diversity in the case of denoising models (if that's what you mean), because you never specifically train the model to map $\mathbf{x}_T$ to $\mathbf{x}_0$. Instead, you are training it to slightly improve the quality of corrupted samples. This denoising itself is stochastic. Therefore over multiple denoising steps, the outcome can vary greatly. For an example, see Figure 7 in the [Ho et al. paper](https://arxiv.org/pdf/2006.11239.pdf).
Comments.UserDisplayName: Chillston
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-03T18:30:50.957Z
Comments.Text: @Chillston let me pick your brain a bit more, is $q(\textbf{x}_{t-1} | \textbf{x}_t , \textbf{x}_0) = \mathcal{N}(\tilde{\bf{\mu}}(\textbf{x}_t , \textbf{x}_0) \, , \, \tilde{\beta}_t \textbf{I})$  (the posteriori we could obtain conditioning on $\textbf{x}_0$) the best possible reverse process we could hope to achieve or can we somehow learn something better? It seems to me we can't hope to learn a $p_{\Theta}( \textbf{x}_{t-1} | \textbf{x}_t) )$ that is better.
Comments.UserDisplayName: Monolite
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-04T12:04:07.850Z
Comments.Text: @Monolite Hmm, the way I understand it is that calculating the posterior $q(\mathbf{x}_{t-1} | \mathbf{x}_t)$ is infeasible because you cannot *guess* the noise $\epsilon$ of the term $\mathbf{x}_{t-1} + \epsilon = \mathbf{x}_{t}$. But when you condition on $\mathbf{x}_{0}$ then you drastically narrow down the distribution of $\epsilon$. This reduces a lot of the variance of the prior guess.
Comments.UserDisplayName: Chillston
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-04T12:06:39.733Z
Comments.Text: However, as the blog author also states, conditioning on $\mathbf{x}_0$ will result in reconstructing $\mathbf{x}_0$, which is close to the behavior of a VAE. So if you only want to reconstruct, then this might be the best thing to do. If you want to generate data from the manifold, it is not the best option (in my understanding).
Comments.UserDisplayName: Chillston
Tags: <generative-model><image-generation>
Title: What is the exact role of model $p_\theta$ in diffusion models for the reverse process?
UserDisplayName: James Arten
ThreadText: # What is the exact role of model $p_\theta$ in diffusion models for the reverse process?
  
  (Asked by: username_0 on 2022-06-24)
  
  I'm reading this interesting [blog post](https://lilianweng.github.io/posts/2021-07-11-diffusion-models/) explaining *diffusion probabilistic models* and trying to understand the following.
  
  In order to compute the *reverse process*, we need to consider the posterior distribution $q(\textbf{x}\_{t-1} | \textbf{x}\_t)$ which is said to be **intractable**\*
  
  > because **it needs to use the entire dataset** and therefore we need to learn a model $p\_\theta$ to approximate these conditional probabilities in order to run the reverse diffusion process.
  
  If we use *Bayes theorem* we have
  
  $$q(\textbf{x}\_{t-1} | \textbf{x}\_t) = \frac{q(\textbf{x}\_t |\textbf{x}\_{t-1})q(\textbf{x}\_{t-1})}{q(\textbf{x}\_t)}$$
  
  I understand that indeed we don't have any prior knowledge of $q(\textbf{x}\_{t-1})$ or $q(\textbf{x}\_t)$ since this would mean already having the distribution we are trying to estimate. Is this correct?
  
  The above posterior becomes **tractable** when conditioned on $\textbf{x}\_0$ and we obtain
  
  $$q(\textbf{x}\_{t-1} | \textbf{x}\_t , \textbf{x}\_0) = \mathcal{N}(\tilde{\bf{\mu}}(\textbf{x}\_t , \textbf{x}\_0) \, , \, \tilde{\beta}\_t \textbf{I})$$
  
  So, apparently, we obtain a **posterior** that can be calculated in **closed form** when we condition on the original data $\textbf{x}\_0$. At this point, I don't understand the role of the model $p\_\theta$ : why do we need to tune the parameters of a model if we can already obtain our posterior?
  
  **Tags:** generative-model, image-generation
  > **Comments:**
  > **username_1**: Don't know much about diffusion models, but I think this iterative conversion of $\mathbf{x}_0$ to a gaussian is only being done to then train a model to reverse the process. So in my understanding, the whole idea is to not use $\mathbf{x}_0$, otherwise, why would you want to add noise to $\mathbf{x}_0$ in the first place? This might be comparable to the encoding and decoding process of variational autoencoders. The encoding makes the input noisier and the decoder reconstructs the input. Here, you only condition the encoder on $\mathbf{x}_0$, but not the decoder directly.  
  > **username_2**: @Chillston Thank you greatly for your answer! So when we are optimizing the lower bound on the negative log-likelihood even there we are not using $\mathbf{x}_0$  correct?  Since I have you here maybe you can also tell me the intuition behind setting up a variational inference problem with a fixed approximate posterior, I thought the whole point of variational inference was to allow the posterior to belong to a big enough (well-behaved) class so as to get as close as possible to the true posterior in an efficient way.  
  > **username_1**: I hope I understand you right, I really didn't look into diffusion models too deep, yet. How I understand this is: You ultimately want a model that denoises a given random sample and maps it onto the manifold of your data. Because it is not feasible to model $p_{\Theta} : X_T \rightarrow X_0$ directly, the model learns a single denoising step $p_{\Theta} : X_{t} \rightarrow X_{t+1}$ ($0 \leq t \leq T$). By iteratively applying the model you arrive at $X_0$ after $T$ iterations. Thus given a noisy sample $x_t$, the model predicts the noise that was added to $\mathbf{x}_{t-1}$.  
  > **username_1**: So the (simplified) training objective is actually the MSE between predicted and actual noise (s. Equation 14 [Ho et al.](https://arxiv.org/pdf/2006.11239.pdf)). So I'd say you need $\mathbf{x}_0$ to optimize the variational bound.  
  > **username_1**: Regarding the second part: In my intuition, having a fixed posterior doesn't limit diversity in the case of denoising models (if that's what you mean), because you never specifically train the model to map $\mathbf{x}_T$ to $\mathbf{x}_0$. Instead, you are training it to slightly improve the quality of corrupted samples. This denoising itself is stochastic. Therefore over multiple denoising steps, the outcome can vary greatly. For an example, see Figure 7 in the [Ho et al. paper](https://arxiv.org/pdf/2006.11239.pdf).  
  > **username_2**: @Chillston let me pick your brain a bit more, is $q(\textbf{x}_{t-1} | \textbf{x}_t , \textbf{x}_0) = \mathcal{N}(\tilde{\bf{\mu}}(\textbf{x}_t , \textbf{x}_0) \, , \, \tilde{\beta}_t \textbf{I})$  (the posteriori we could obtain conditioning on $\textbf{x}_0$) the best possible reverse process we could hope to achieve or can we somehow learn something better? It seems to me we can't hope to learn a $p_{\Theta}( \textbf{x}_{t-1} | \textbf{x}_t) )$ that is better.  
  > **username_1**: @Monolite Hmm, the way I understand it is that calculating the posterior $q(\mathbf{x}_{t-1} | \mathbf{x}_t)$ is infeasible because you cannot *guess* the noise $\epsilon$ of the term $\mathbf{x}_{t-1} + \epsilon = \mathbf{x}_{t}$. But when you condition on $\mathbf{x}_{0}$ then you drastically narrow down the distribution of $\epsilon$. This reduces a lot of the variance of the prior guess.  
  > **username_1**: However, as the blog author also states, conditioning on $\mathbf{x}_0$ will result in reconstructing $\mathbf{x}_0$, which is close to the behavior of a VAE. So if you only want to reconstruct, then this might be the best thing to do. If you want to generate data from the manifold, it is not the best option (in my understanding).  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2022-09-25. Score: 2
  
  I am also learning diffusion models and would like to give some information.
  
  > At this point, I don't understand the role of the model $p\_\theta$
  
  To clear a bit: $p\_\theta$ is just another annotation for U-net and the role is receiving ($x\_t$,$t$) (sometimes also receives classifier $y$) and predicts $x\_0$ OR $x\_{t-1}$ depending on different papers. So at the end of the day, to synthesize new data, given a noisy (usually Gaussian) image, U-net can iteratively predict $x\_0$ better - check out algorithm 2 in the DDPM paper (2020).
  
  Your question about the posterior might be answered in more detail here: [Diffusion Models | Paper Explanation | Math Explained - YouTube](https://www.youtube.com/watch?v=HoKDTa5jHvg)
  
  Check video time around **18:00** that explains a bit more information regarding $x\_0$ guided process in the optimization of lower boundary.
  
  > **Comments:**
  > **username_4**: Why is noise added at each timestamp during the reverse process?  
  > **username_3**: @NathanB I am assuming you mean the DDIM diffusion way. DDIM is predicting x_0 and adding the noise back to x_t-1 during reverse. From my understanding, this strategy is turning the diffusion process from markovian to non-markovian. The benefits of DDIM include faster generation, better sampling strategy, etc. I would recommend the cold diffusion paper and the original DDIM paper for a deeper understanding.  
  > **username_4**: What will happen if you don't add some noise in the reverse process at every step? After all, you start from a random noise image, so you already have the randomness needed for the sampling, why add more noise every step backwards?  
  > **username_3**: @NathanB you can do it without adding noise during sampling. It's just different sampling strategies. In the original DDPM paper, they are not adding noise when sampling backward.  
  > **username_4**: Is the purpose to basically add more 'randomness' to the process so that the result will be more 'creative'?  
  > **username_3**: @NathanB I can't give you the conclusion that it's adding more "randomness" to be more "creative" as I haven't fully researched it enough. However, the main purpose includes changing non-markovian to markovian, which allows faster generation. Different sampling strategies have different pros and cons, I highly suggest reading papers like cold diffusion, stable diffusion, ILVR, etc. to get a deeper intuition of those sampling strategies.  
  > **username_4**: Still can't find an answer to the question: Why in the DDIM a noise is added every step of the reverse process, and if it makes it better.  
  
  ---
  
  ### Answer by username_5 on 2022-11-19. Score: 0
  
  You do not yet have $\mathbf{x}\_0$ during sampling (not training). That's why you need to approximate $q(\mathbf{x}\_{t−1}|\mathbf{x}\_t, \mathbf{x}\_0)$ with $p\_{\theta}(\mathbf{x}\_{t−1}|\mathbf{x}\_t)$ via variational inference such as KL divergence. After training with good data, this should produce an approximation of $\mathbf{x}\_0$.
  
  
  ---
  
  