Answers.Body: <p>while we set <span class="math-container">$R$</span> to be independent from <span class="math-container">$x_{t-1}$</span> in the calculation of <span class="math-container">$x_t$</span>, we no long have independency between <span class="math-container">$x_t$</span> and <span class="math-container">$R$</span>, and I guess this gives rise to the confusion.</p>
  <p>For simplicity, let us consider two independent normal variable <span class="math-container">$x, \varepsilon \sim \mathcal{N}\left(0, 1\right)$</span>, and let us set <span class="math-container">$y = ax + b\varepsilon$</span>. Then we have
  <span class="math-container">$$
  y|x = \mathcal{N}\left(y; ax, b^2\right),
  $$</span>
  (For more detail on conditional normal, see this link: <a href="https://statproofbook.github.io/P/mvn-cond" rel="nofollow noreferrer">https://statproofbook.github.io/P/mvn-cond</a>)
  which is similar to your conclusion.</p>
  <p>But we have
  <span class="math-container">$$
  x|y = \mathcal{N}\left(x; \frac{ay}{a^2 + b^2}, \frac{b^2}{a^2 + b^2}\right)
  $$</span></p>
  <p>(You may also find this link: <a href="https://www.statlect.com/probability-distributions/normal-distribution-linear-combinations" rel="nofollow noreferrer">https://www.statlect.com/probability-distributions/normal-distribution-linear-combinations</a> helpful in computing the linear combination of multivariable normal)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-17T15:27:08.840Z
Answers.LastActivityDate: 2023-01-17T15:27:08.840Z
Answers.UserDisplayName: Yi Huang
Answers.Body: <p>Consider the simple case where <span class="math-container">$x_{t-1} = c$</span> is deterministic and <span class="math-container">$\beta &gt; 0$</span>. Then the forward process is given by <span class="math-container">$$q(x_t|x_{t-1}) \sim \mathcal{N}(x_t; \sqrt{1 - \beta}c, \beta I)$$</span> But the backward process would definitely not be given by <span class="math-container">$$q(x_{t-1}|x_t) \sim \mathcal{N}(x_t; (1 - \beta)^{-1/2}x_t, \frac{\beta}{1 - \beta} I)$$</span>
  as <span class="math-container">$x_{t-1}$</span> is deterministic.</p>
  <p>Of course ironically in the example I presented, the backward process is trivial, but it shows you can't just &quot;solve for <span class="math-container">$x_{t-1}$</span>&quot;.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-22T21:52:48.850Z
Answers.LastActivityDate: 2023-01-22T21:52:48.850Z
Answers.UserDisplayName: Jack Ding
Body: <p>In explanations of denoising diffusion models it is stated that <span class="math-container">$q(x_{t-1}|x_t)$</span> is intractable. This is often justified via Bayes' rule, i.e.
  <span class="math-container">$$
  q(x_{t-1}|x_t) \propto q(x_t|x_{t-1})q(x_{t-1})
  $$</span>
  and the marginal <span class="math-container">$q(x_{t-1})$</span> is unknown. But I'm confused. We know that
  <span class="math-container">$$
  x_t = \sqrt{1-\beta_t}x_{t-1}+\sqrt{\beta_t}E,\quad E\sim\mathcal{N}(0,I)
  $$</span>
  therefore we can solve this equation for <span class="math-container">$x_{t-1}$</span>:
  <span class="math-container">\begin{align}
  x_{t-1} &amp;= (1-\beta_t)^{-1/2}x_t - \sqrt{\frac{\beta_t}{1-\beta_t}}E
  \\
  &amp;= (1-\beta_t)^{-1/2}x_t + \sqrt{\frac{\beta_t}{1-\beta_t}}R,\quad R\sim\mathcal{N}(0,I).
  \end{align}</span>
  Thus
  <span class="math-container">$$
  q(x_{t-1}|x_t) = \mathcal{N}(x_{t-1};(1-\beta_t)^{-1/2}x_t,\frac{\beta_t}{1-\beta_t}I).
  $$</span>
  This is simple as can be. If this is true, there is no point in parameterizing the reverse distribution with neural nets and we don't need <span class="math-container">$\beta_t$</span> to be small etc. What am I missing?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-03-19T19:48:04.473Z
Comments.Text: Hi, did you end up finding the solution to this?
Comments.UserDisplayName: cabralpinto
Tags: <machine-learning><generative-model><diffusion-models><image-denoising>
Title: Reverse Distribution in Denoising Diffusion Models is Simple
UserDisplayName: Patrick Johnstone
ThreadText: # Reverse Distribution in Denoising Diffusion Models is Simple
  
  (Asked by: username_0 on 2023-01-11)
  
  In explanations of denoising diffusion models it is stated that $q(x\_{t-1}|x\_t)$ is intractable. This is often justified via Bayes' rule, i.e.
  $$
  q(x\_{t-1}|x\_t) \propto q(x\_t|x\_{t-1})q(x\_{t-1})
  $$
  and the marginal $q(x\_{t-1})$ is unknown. But I'm confused. We know that
  $$
  x\_t = \sqrt{1-\beta\_t}x\_{t-1}+\sqrt{\beta\_t}E,\quad E\sim\mathcal{N}(0,I)
  $$
  therefore we can solve this equation for $x\_{t-1}$:
  \begin{align}
  x\_{t-1} &= (1-\beta\_t)^{-1/2}x\_t - \sqrt{\frac{\beta\_t}{1-\beta\_t}}E
  \\
  &= (1-\beta\_t)^{-1/2}x\_t + \sqrt{\frac{\beta\_t}{1-\beta\_t}}R,\quad R\sim\mathcal{N}(0,I).
  \end{align}
  Thus
  $$
  q(x\_{t-1}|x\_t) = \mathcal{N}(x\_{t-1};(1-\beta\_t)^{-1/2}x\_t,\frac{\beta\_t}{1-\beta\_t}I).
  $$
  This is simple as can be. If this is true, there is no point in parameterizing the reverse distribution with neural nets and we don't need $\beta\_t$ to be small etc. What am I missing?
  
  **Tags:** machine-learning, generative-model, diffusion-models, image-denoising
  > **Comments:**
  > **username_1**: Hi, did you end up finding the solution to this?  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2023-01-17. Score: 1
  
  while we set $R$ to be independent from $x\_{t-1}$ in the calculation of $x\_t$, we no long have independency between $x\_t$ and $R$, and I guess this gives rise to the confusion.
  
  For simplicity, let us consider two independent normal variable $x, \varepsilon \sim \mathcal{N}\left(0, 1\right)$, and let us set $y = ax + b\varepsilon$. Then we have
  $$
  y|x = \mathcal{N}\left(y; ax, b^2\right),
  $$
  (For more detail on conditional normal, see this link: <https://statproofbook.github.io/P/mvn-cond>)
  which is similar to your conclusion.
  
  But we have
  $$
  x|y = \mathcal{N}\left(x; \frac{ay}{a^2 + b^2}, \frac{b^2}{a^2 + b^2}\right)
  $$
  
  (You may also find this link: <https://www.statlect.com/probability-distributions/normal-distribution-linear-combinations> helpful in computing the linear combination of multivariable normal)
  
  
  ---
  
  ### Answer by username_3 on 2023-01-22. Score: 0
  
  Consider the simple case where $x\_{t-1} = c$ is deterministic and $\beta > 0$. Then the forward process is given by $$q(x\_t|x\_{t-1}) \sim \mathcal{N}(x\_t; \sqrt{1 - \beta}c, \beta I)$$ But the backward process would definitely not be given by $$q(x\_{t-1}|x\_t) \sim \mathcal{N}(x\_t; (1 - \beta)^{-1/2}x\_t, \frac{\beta}{1 - \beta} I)$$
  as $x\_{t-1}$ is deterministic.
  
  Of course ironically in the example I presented, the backward process is trivial, but it shows you can't just "solve for $x\_{t-1}$".
  
  
  ---
  
  