Answers.Body: <p>Indeed the notations here are a little involved and confusing since there exist multiple joint conditional random variables here. First please note here all involved conditional random variables are <em>all</em> conditioned only on the fixed value of the input data <span class="math-container">$x_0$</span>, so we can tentatively and effectively ignore it when considering other (conditional) random variables. Next let's look at the second term in your asked line (44): <span class="math-container">$$\mathbb{E}_{q(x_{T−1}, x_T | x_0)} [\log \frac{p(x_T)}{q(x_T | x_{T −1})}]$$</span></p>
  <p>Notice compared to the usual KL divergence, there's an additional fixed conditioning value of <span class="math-container">$x_{T−1}$</span> as given data involved in the posterior <span class="math-container">$q$</span>-distribution of the random variable <span class="math-container">$x_T$</span> in above term. And since the random variable <span class="math-container">$x_{T−1}$</span> is taken <em>jointly</em> with the random variable <span class="math-container">$x_T$</span> given <span class="math-container">$x_0$</span> over the expectation in above term, we have to take expectation wrt the random variable <span class="math-container">$x_{T−1}$</span> <em>after</em> rewriting the above term as KL divergence. With these ideas we can expand the above term as:
  <span class="math-container">$$\mathbb{E}_{x_{T−1}|x_0 \sim q(x_{T−1}| x_0)}[\mathbb{E}_{x_T|x_{T-1}, x_0 \sim q(x_T|x_{T-1}, x_0)} [\log \frac{p(x_T)}{q(x_T | x_{T −1})}]] \\= -\mathbb{E}_{q(x_{T−1}| x_0)}[\mathbb{E}_{q(x_T|x_{T-1})} [\log \frac{q(x_T | x_{T −1})}{p(x_T)}]] \\= -\mathbb{E}_{q(x_{T−1}| x_0)}[D_{KL}(q(x_T | x_{T −1})||p(x_T))]$$</span></p>
  <p>Similar process can be applied to the third term to explain the additional expectation operator of the rewritten KL divergence.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-02T05:47:32.863Z
Answers.LastActivityDate: 2024-09-09T07:50:16.553Z
Answers.LastEditDate: 2024-09-09T07:50:16.553Z
Answers.UserDisplayName: cinch
Body: <p>I am trying to read through the proof of ELBO for diffusion models on pg. 8 of this <a href="https://arxiv.org/pdf/2208.11970" rel="nofollow noreferrer">paper</a>. However, I do not see how the author arrived at Eqn (45) from Eqn (44). Specifically, I do not know how they simplified the equation by rewriting it in terms of the KL divergence. I do know the formula for KL divergence but it does not seem like a direct application in this case. Any help is appreciated. I have included an image below of the proof for convenience.</p>
  <p><a href="https://i.sstatic.net/0kE1Tm8C.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/0kE1Tm8C.jpg" alt="enter image description here" /></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-09-04T07:12:06.567Z
Comments.Text: Not really an answer, but same question asked [here](https://stats.stackexchange.com/q/653800/429415)
Comments.UserDisplayName: randomforest42
Tags: <deep-learning><variational-autoencoder><diffusion-models><evidence-lower-bound><kl-divergence>
Title: Deriving ELBO for Diffusion Models
UserDisplayName: Nikhil Sridhar
ThreadText: # Deriving ELBO for Diffusion Models
  
  (Asked by: username_0 on 2024-08-23)
  
  I am trying to read through the proof of ELBO for diffusion models on pg. 8 of this [paper](https://arxiv.org/pdf/2208.11970). However, I do not see how the author arrived at Eqn (45) from Eqn (44). Specifically, I do not know how they simplified the equation by rewriting it in terms of the KL divergence. I do know the formula for KL divergence but it does not seem like a direct application in this case. Any help is appreciated. I have included an image below of the proof for convenience.
  
  [![enter image description here](https://i.sstatic.net/0kE1Tm8C.jpg)](https://i.sstatic.net/0kE1Tm8C.jpg)
  
  **Tags:** deep-learning, variational-autoencoder, diffusion-models, evidence-lower-bound, kl-divergence
  > **Comments:**
  > **username_1**: Not really an answer, but same question asked [here](https://stats.stackexchange.com/q/653800/429415)  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2024-09-02. Score: 0 | Accepted answer
  
  Indeed the notations here are a little involved and confusing since there exist multiple joint conditional random variables here. First please note here all involved conditional random variables are *all* conditioned only on the fixed value of the input data $x\_0$, so we can tentatively and effectively ignore it when considering other (conditional) random variables. Next let's look at the second term in your asked line (44): $$\mathbb{E}\_{q(x\_{T−1}, x\_T | x\_0)} [\log \frac{p(x\_T)}{q(x\_T | x\_{T −1})}]$$
  
  Notice compared to the usual KL divergence, there's an additional fixed conditioning value of $x\_{T−1}$ as given data involved in the posterior $q$-distribution of the random variable $x\_T$ in above term. And since the random variable $x\_{T−1}$ is taken *jointly* with the random variable $x\_T$ given $x\_0$ over the expectation in above term, we have to take expectation wrt the random variable $x\_{T−1}$ *after* rewriting the above term as KL divergence. With these ideas we can expand the above term as:
  $$\mathbb{E}\_{x\_{T−1}|x\_0 \sim q(x\_{T−1}| x\_0)}[\mathbb{E}\_{x\_T|x\_{T-1}, x\_0 \sim q(x\_T|x\_{T-1}, x\_0)} [\log \frac{p(x\_T)}{q(x\_T | x\_{T −1})}]] \\= -\mathbb{E}\_{q(x\_{T−1}| x\_0)}[\mathbb{E}\_{q(x\_T|x\_{T-1})} [\log \frac{q(x\_T | x\_{T −1})}{p(x\_T)}]] \\= -\mathbb{E}\_{q(x\_{T−1}| x\_0)}[D\_{KL}(q(x\_T | x\_{T −1})||p(x\_T))]$$
  
  Similar process can be applied to the third term to explain the additional expectation operator of the rewritten KL divergence.
  
  
  ---
  
  