Answers.Body: <p>One of my colleagues gave me a solution. (Thanks Ken!)</p>
  <p><span class="math-container">$$
  {\begin{aligned}
  \sigma_{t|s}^2 &amp;= 1 - \alpha_{t|s}^2 \\
  &amp;= 1 - {\alpha_t^2 \over \alpha_s^2} \\
  &amp;= {\alpha_s^2 - \alpha_t^2 \over \alpha_s^2} \\
  &amp;= {(1 - \sigma_s^2)-(1-\sigma_t^2) \over \alpha_s^2} \\
  &amp;= {\sigma_t^2 - \sigma_s^2 \over \alpha_s^2} \\
  &amp;= {\sigma_t^2 (\alpha_s^2 + \sigma_s^2) - \sigma_s^2 (\alpha_t^2 + \sigma_t^2) \over \alpha_s^2} \\
  &amp;= {\sigma_t^2 \alpha_s^2 - \sigma_s^2 \alpha_t^2 \over \alpha_s^2} \\
  &amp;= \left(1 -{\sigma_s^2 \alpha_t^2 \over \alpha_s^2 \sigma_t^2}\right) \sigma_t^2 \\
  &amp;= \left(1 -{1 \over e^{\lambda_s}}\cdot e^{\lambda_t}\right) \sigma_t^2 \\
  &amp;= \left(1 - e^{\lambda_t - \lambda_s}\right) \sigma_t^2 \\
  \end{aligned}}
  $$</span>
  given
  <span class="math-container">$$
  \alpha_t^2 + \sigma_t^2 = 1 \\
  \alpha_{t|s}^2 + \sigma_{t|s}^2 = 1 \\
  e^{\lambda_t} = {\alpha_t^2 \over \sigma_t^2}
  $$</span>
  .</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-14T11:14:30.810Z
Answers.LastActivityDate: 2023-02-14T11:14:30.810Z
Answers.UserDisplayName: hotohoto
Answers.Body: <p>There is another derivation:
  <span class="math-container">$$
  {\begin{aligned}
  \sigma_{t|s}^2 &amp;:= \sigma_t^2 - {\alpha_{t}^2 \over \alpha_s^2} \sigma_s^2 \\
  &amp;= \left( 1 - {\alpha_t^2 \sigma_s^2\over \sigma_t^2\alpha_s^2} \right) \sigma_t^2\\
  &amp;= \left(1 -e^{\lambda_t}\cdot {1 \over e^{\lambda_s}}\right) \sigma_t^2 \\
  &amp;= \left(1 - e^{\lambda_t - \lambda_s}\right) \sigma_t^2 \\
  \end{aligned}}
  $$</span></p>
  <p>given
  <span class="math-container">$$
  e^{\lambda_t} = {\alpha_t^2 \over \sigma_t^2}
  $$</span>
  .</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-15T12:33:37.067Z
Answers.LastActivityDate: 2023-02-15T12:33:37.067Z
Answers.UserDisplayName: hotohoto
Body: <p>Regarding Eq. (1) in <a href="https://arxiv.org/abs/2202.00512" rel="nofollow noreferrer">Progressive Distillation for Fast Sampling of Diffusion Models</a>,
  <span class="math-container">$$
  q(\mathbf{z}_t|\mathbf{z}_s) = \mathcal{N}(\mathbf{z}_t; (\alpha_t/\alpha_s)\mathbf{z}_s, \sigma _{t|s}^2 \mathbf{I})
  \tag{1}
  $$</span>
  , it says <span class="math-container">$\sigma_{t|s}^2 = (1 - e^{\lambda_t - \lambda_s})\sigma_t^2$</span>.</p>
  <p>I believe <span class="math-container">$\sigma_{t|s}$</span> is from DDPM's <span class="math-container">$\beta$</span>, then it's equivalent to <span class="math-container">$1 - \alpha_t^2 / \alpha_s^2$</span> here. Note that <span class="math-container">$\lambda_t = \log (\alpha_t^2 / \sigma_t^2)$</span>. As far as I understand, <span class="math-container">$\sigma_t^2 + \alpha_t^2 = 1$</span>. Maybe I'm taking something wrong.</p>
  <p>Anyway, how could <span class="math-container">$\sigma_{t|s}^2 = (1 - e^{\lambda_t - \lambda_s})\sigma_t^2$</span> be derived from <span class="math-container">$1 - \alpha_t^2 / \alpha_s^2$</span>?</p>
  <p>Some more notes.</p>
  <ul>
  <li>More papers are using the same expression.
  <ul>
  <li><a href="https://arxiv.org/abs/2207.12598" rel="nofollow noreferrer">Classifier-Free Diffusion Guidance</a></li>
  <li><a href="https://arxiv.org/abs/2204.03458" rel="nofollow noreferrer">Video Diffusion Models</a></li>
  <li><a href="https://arxiv.org/abs/2210.02303" rel="nofollow noreferrer">Imagen Video: High Definition Video Generation with Diffusion Models</a></li>
  </ul>
  </li>
  <li>I think the definition of <span class="math-container">$\sigma_{t|s}$</span> Eq. (22) in <a href="https://arxiv.org/abs/2107.00630" rel="nofollow noreferrer">the original VDM paper</a> is straightforward with respect to DDPM's definitions.</li>
  <li>The expression seems to be quite similar to Eq.(32) of VDM.</li>
  </ul>
  
Tags: <diffusion-models>
Title: How to derive the variance of the forward step of Variational Diffusion Models in terms of the log signal-to-noise ratio $\lambda_t$?
UserDisplayName: hotohoto
ThreadText: # How to derive the variance of the forward step of Variational Diffusion Models in terms of the log signal-to-noise ratio $\lambda_t$?
  
  (Asked by: username_0 on 2023-02-14)
  
  Regarding Eq. (1) in [Progressive Distillation for Fast Sampling of Diffusion Models](https://arxiv.org/abs/2202.00512),
  $$
  q(\mathbf{z}\_t|\mathbf{z}\_s) = \mathcal{N}(\mathbf{z}\_t; (\alpha\_t/\alpha\_s)\mathbf{z}\_s, \sigma \_{t|s}^2 \mathbf{I})
  \tag{1}
  $$
  , it says $\sigma\_{t|s}^2 = (1 - e^{\lambda\_t - \lambda\_s})\sigma\_t^2$.
  
  I believe $\sigma\_{t|s}$ is from DDPM's $\beta$, then it's equivalent to $1 - \alpha\_t^2 / \alpha\_s^2$ here. Note that $\lambda\_t = \log (\alpha\_t^2 / \sigma\_t^2)$. As far as I understand, $\sigma\_t^2 + \alpha\_t^2 = 1$. Maybe I'm taking something wrong.
  
  Anyway, how could $\sigma\_{t|s}^2 = (1 - e^{\lambda\_t - \lambda\_s})\sigma\_t^2$ be derived from $1 - \alpha\_t^2 / \alpha\_s^2$?
  
  Some more notes.
  
  * More papers are using the same expression.
    + [Classifier-Free Diffusion Guidance](https://arxiv.org/abs/2207.12598)
    + [Video Diffusion Models](https://arxiv.org/abs/2204.03458)
    + [Imagen Video: High Definition Video Generation with Diffusion Models](https://arxiv.org/abs/2210.02303)
  * I think the definition of $\sigma\_{t|s}$ Eq. (22) in [the original VDM paper](https://arxiv.org/abs/2107.00630) is straightforward with respect to DDPM's definitions.
  * The expression seems to be quite similar to Eq.(32) of VDM.
  
  **Tags:** diffusion-models
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_0 on 2023-02-14. Score: 1
  
  One of my colleagues gave me a solution. (Thanks Ken!)
  
  $$
  {\begin{aligned}
  \sigma\_{t|s}^2 &= 1 - \alpha\_{t|s}^2 \\
  &= 1 - {\alpha\_t^2 \over \alpha\_s^2} \\
  &= {\alpha\_s^2 - \alpha\_t^2 \over \alpha\_s^2} \\
  &= {(1 - \sigma\_s^2)-(1-\sigma\_t^2) \over \alpha\_s^2} \\
  &= {\sigma\_t^2 - \sigma\_s^2 \over \alpha\_s^2} \\
  &= {\sigma\_t^2 (\alpha\_s^2 + \sigma\_s^2) - \sigma\_s^2 (\alpha\_t^2 + \sigma\_t^2) \over \alpha\_s^2} \\
  &= {\sigma\_t^2 \alpha\_s^2 - \sigma\_s^2 \alpha\_t^2 \over \alpha\_s^2} \\
  &= \left(1 -{\sigma\_s^2 \alpha\_t^2 \over \alpha\_s^2 \sigma\_t^2}\right) \sigma\_t^2 \\
  &= \left(1 -{1 \over e^{\lambda\_s}}\cdot e^{\lambda\_t}\right) \sigma\_t^2 \\
  &= \left(1 - e^{\lambda\_t - \lambda\_s}\right) \sigma\_t^2 \\
  \end{aligned}}
  $$
  given
  $$
  \alpha\_t^2 + \sigma\_t^2 = 1 \\
  \alpha\_{t|s}^2 + \sigma\_{t|s}^2 = 1 \\
  e^{\lambda\_t} = {\alpha\_t^2 \over \sigma\_t^2}
  $$
  .
  
  
  ---
  
  ### Answer by username_0 on 2023-02-15. Score: 1
  
  There is another derivation:
  $$
  {\begin{aligned}
  \sigma\_{t|s}^2 &:= \sigma\_t^2 - {\alpha\_{t}^2 \over \alpha\_s^2} \sigma\_s^2 \\
  &= \left( 1 - {\alpha\_t^2 \sigma\_s^2\over \sigma\_t^2\alpha\_s^2} \right) \sigma\_t^2\\
  &= \left(1 -e^{\lambda\_t}\cdot {1 \over e^{\lambda\_s}}\right) \sigma\_t^2 \\
  &= \left(1 - e^{\lambda\_t - \lambda\_s}\right) \sigma\_t^2 \\
  \end{aligned}}
  $$
  
  given
  $$
  e^{\lambda\_t} = {\alpha\_t^2 \over \sigma\_t^2}
  $$
  .
  
  
  ---
  
  