Answers.Body: <p>Using the reparameterization trick that tells us:
  <span class="math-container">$$\begin{aligned}
  \mathbf{z} &amp;\sim \mathcal{N}(\mathbf{z}; \boldsymbol{\mu}, \boldsymbol{\sigma}\boldsymbol{I}) \\
  \mathbf{z} &amp;= \boldsymbol{\mu} + \boldsymbol{\sigma} \odot \boldsymbol{\epsilon} \text{, where } \boldsymbol{\epsilon} \sim \mathcal{N}(0, \boldsymbol{I})
  \end{aligned}$$</span>
  If we define <span class="math-container">$\alpha_t = 1 - \beta_t$</span> and <span class="math-container">$\bar{\alpha}_t = \prod_{i=1}^t \alpha_i$</span>:
  <span class="math-container">$$
  \begin{aligned}
  \mathbf{x}_t 
  &amp;= \sqrt{\alpha_t}\mathbf{x}_{t-1} + \sqrt{1 - \alpha_t}\boldsymbol{\epsilon}_{t-1}  \\
  &amp;= \sqrt{\alpha_t}(\sqrt{\alpha_{t-1}} \mathbf{x}_{t-2} + \sqrt{1 - \alpha_{t-1}}\boldsymbol{\epsilon}_{t-2}) + \sqrt{1 - \alpha_t} \boldsymbol{\epsilon}_{t-1} \\
  &amp;= \sqrt{\alpha_t \alpha_{t-1}}\mathbf{x}_{t-2} + \sqrt{\alpha_t(1 - \alpha_{t-1})}\boldsymbol{\epsilon}_{t-2} + \sqrt{1 - \alpha_{t}}\boldsymbol{\epsilon}_{t-1} \\
  &amp;= \sqrt{\alpha_t \alpha_{t-1}}\mathbf{x}_{t-2} + \sqrt{\alpha_t(1 - \alpha_{t-1})+(1 - \alpha_{t})}\bar{\boldsymbol{\epsilon}}_{t-2} \\
  &amp;= \sqrt{\alpha_t \alpha_{t-1}}\mathbf{x}_{t-2} + \sqrt{1-\alpha_t \alpha_{t-1}}\bar{\boldsymbol{\epsilon}}_{t-2} \\
  &amp;= \dots \\
  &amp;= \sqrt{\bar{\alpha}_t}\mathbf{x}_0 + \sqrt{1 - \bar{\alpha}_t}\boldsymbol{\epsilon} \\
  q(\mathbf{x}_t \vert \mathbf{x}_0) &amp;= \mathcal{N}(\mathbf{x}_t; \sqrt{\bar{\alpha}_t} \mathbf{x}_0, (1 - \bar{\alpha}_t)\mathbf{I})
  \end{aligned}
  $$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-14T13:04:26.280Z
Answers.LastActivityDate: 2023-04-14T13:04:26.280Z
Answers.UserDisplayName: Deb
Body: <p>So I'm watching <a href="https://youtu.be/cS6JQpEY9cs?t=835" rel="nofollow noreferrer">this video tutorial from CVPR this year on diffusion models</a>, and I am confused by the variance term in the distribution on the left on the video. I understand that in the forward process, we can track intermediate distributions</p>
  <p><span class="math-container">$$q(\mathbf{x}_t|\mathbf{x}_{t-1}) = \mathcal{N}(\mathbf{x}_t;\sqrt{1-\beta_t}\mathbf{x}_{t-1},\beta_t\mathbf{I})$$</span></p>
  <p>And that the joint distributions of all the intermediate steps conditioned on the input is given by</p>
  <p><span class="math-container">$$q(\mathbf{x}_{1:T}|\mathbf{x}_{0}) = \prod_{t=1}^Tq(\mathbf{x}_t|\mathbf{x}_{t-1}).$$</span></p>
  <p>If we define <span class="math-container">$\bar{\alpha} = \prod_{s = 1}^t(1 - \beta_t),$</span> then we are supposed to derive the diffusion kernel</p>
  <p><span class="math-container">$$q(\mathbf{x}_t|\mathbf{x}_{0}) = \mathcal{N}(\mathbf{x}_t;\sqrt{\bar{\alpha}}\mathbf{x}_0,(1 - \bar{\alpha}_t)\mathbf{I})$$</span></p>
  <p>I can definitely see why the mean is what it is, but I'm having a hard time seeing where we get the variance from. How is the variance derived?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-11-13T15:53:32.177Z
Comments.Text: While I think this question fits here, you might have better chances of an answer in stats.stackexchange.com
Comments.UserDisplayName: Dr. Snoopy
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-11-19T20:12:08.713Z
Comments.Text: https://math.stackexchange.com/a/4476159/574215 This answer exactly derives both the mean and variance.
Comments.UserDisplayName: lkjsfkshd
Tags: <computer-vision><generative-model><probability-distribution><variance><diffusion-models>
Title: How is the variance for a diffusion kernel derived for a diffusion model?
UserDisplayName: Cynthia Kim
ThreadText: # How is the variance for a diffusion kernel derived for a diffusion model?
  
  (Asked by: username_0 on 2022-11-12)
  
  So I'm watching [this video tutorial from CVPR this year on diffusion models](https://youtu.be/cS6JQpEY9cs?t=835), and I am confused by the variance term in the distribution on the left on the video. I understand that in the forward process, we can track intermediate distributions
  
  $$q(\mathbf{x}\_t|\mathbf{x}\_{t-1}) = \mathcal{N}(\mathbf{x}\_t;\sqrt{1-\beta\_t}\mathbf{x}\_{t-1},\beta\_t\mathbf{I})$$
  
  And that the joint distributions of all the intermediate steps conditioned on the input is given by
  
  $$q(\mathbf{x}\_{1:T}|\mathbf{x}\_{0}) = \prod\_{t=1}^Tq(\mathbf{x}\_t|\mathbf{x}\_{t-1}).$$
  
  If we define $\bar{\alpha} = \prod\_{s = 1}^t(1 - \beta\_t),$ then we are supposed to derive the diffusion kernel
  
  $$q(\mathbf{x}\_t|\mathbf{x}\_{0}) = \mathcal{N}(\mathbf{x}\_t;\sqrt{\bar{\alpha}}\mathbf{x}\_0,(1 - \bar{\alpha}\_t)\mathbf{I})$$
  
  I can definitely see why the mean is what it is, but I'm having a hard time seeing where we get the variance from. How is the variance derived?
  
  **Tags:** computer-vision, generative-model, probability-distribution, variance, diffusion-models
  
  > **Comments:**
  > **username_1**: While I think this question fits here, you might have better chances of an answer in stats.stackexchange.com  
  > **username_2**: https://math.stackexchange.com/a/4476159/574215 This answer exactly derives both the mean and variance.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2023-04-14. Score: 0
  
  Using the reparameterization trick that tells us:
  $$\begin{aligned}
  \mathbf{z} &\sim \mathcal{N}(\mathbf{z}; \boldsymbol{\mu}, \boldsymbol{\sigma}\boldsymbol{I}) \\
  \mathbf{z} &= \boldsymbol{\mu} + \boldsymbol{\sigma} \odot \boldsymbol{\epsilon} \text{, where } \boldsymbol{\epsilon} \sim \mathcal{N}(0, \boldsymbol{I})
  \end{aligned}$$
  If we define $\alpha\_t = 1 - \beta\_t$ and $\bar{\alpha}\_t = \prod\_{i=1}^t \alpha\_i$:
  $$
  \begin{aligned}
  \mathbf{x}\_t
  &= \sqrt{\alpha\_t}\mathbf{x}\_{t-1} + \sqrt{1 - \alpha\_t}\boldsymbol{\epsilon}\_{t-1} \\
  &= \sqrt{\alpha\_t}(\sqrt{\alpha\_{t-1}} \mathbf{x}\_{t-2} + \sqrt{1 - \alpha\_{t-1}}\boldsymbol{\epsilon}\_{t-2}) + \sqrt{1 - \alpha\_t} \boldsymbol{\epsilon}\_{t-1} \\
  &= \sqrt{\alpha\_t \alpha\_{t-1}}\mathbf{x}\_{t-2} + \sqrt{\alpha\_t(1 - \alpha\_{t-1})}\boldsymbol{\epsilon}\_{t-2} + \sqrt{1 - \alpha\_{t}}\boldsymbol{\epsilon}\_{t-1} \\
  &= \sqrt{\alpha\_t \alpha\_{t-1}}\mathbf{x}\_{t-2} + \sqrt{\alpha\_t(1 - \alpha\_{t-1})+(1 - \alpha\_{t})}\bar{\boldsymbol{\epsilon}}\_{t-2} \\
  &= \sqrt{\alpha\_t \alpha\_{t-1}}\mathbf{x}\_{t-2} + \sqrt{1-\alpha\_t \alpha\_{t-1}}\bar{\boldsymbol{\epsilon}}\_{t-2} \\
  &= \dots \\
  &= \sqrt{\bar{\alpha}\_t}\mathbf{x}\_0 + \sqrt{1 - \bar{\alpha}\_t}\boldsymbol{\epsilon} \\
  q(\mathbf{x}\_t \vert \mathbf{x}\_0) &= \mathcal{N}(\mathbf{x}\_t; \sqrt{\bar{\alpha}\_t} \mathbf{x}\_0, (1 - \bar{\alpha}\_t)\mathbf{I})
  \end{aligned}
  $$
  
  
  ---
  
  