<p>Indeed the notations here are a little involved and confusing since there exist multiple joint conditional random variables here. First please note here all involved conditional random variables are <em>all</em> conditioned only on the fixed value of the input data <span class="math-container">$x_0$</span>, so we can tentatively and effectively ignore it when considering other (conditional) random variables. Next let's look at the second term in your asked line (44): <span class="math-container">$$\mathbb{E}_{q(x_{T−1}, x_T | x_0)} [\log \frac{p(x_T)}{q(x_T | x_{T −1})}]$$</span></p>
<p>Notice compared to the usual KL divergence, there's an additional fixed conditioning value of <span class="math-container">$x_{T−1}$</span> as given data involved in the posterior <span class="math-container">$q$</span>-distribution of the random variable <span class="math-container">$x_T$</span> in above term. And since the random variable <span class="math-container">$x_{T−1}$</span> is taken <em>jointly</em> with the random variable <span class="math-container">$x_T$</span> given <span class="math-container">$x_0$</span> over the expectation in above term, we have to take expectation wrt the random variable <span class="math-container">$x_{T−1}$</span> <em>after</em> rewriting the above term as KL divergence. With these ideas we can expand the above term as:
<span class="math-container">$$\mathbb{E}_{x_{T−1}|x_0 \sim q(x_{T−1}| x_0)}[\mathbb{E}_{x_T|x_{T-1}, x_0 \sim q(x_T|x_{T-1}, x_0)} [\log \frac{p(x_T)}{q(x_T | x_{T −1})}]] \\= -\mathbb{E}_{q(x_{T−1}| x_0)}[\mathbb{E}_{q(x_T|x_{T-1})} [\log \frac{q(x_T | x_{T −1})}{p(x_T)}]] \\= -\mathbb{E}_{q(x_{T−1}| x_0)}[D_{KL}(q(x_T | x_{T −1})||p(x_T))]$$</span></p>
<p>Similar process can be applied to the third term to explain the additional expectation operator of the rewritten KL divergence.</p>

CC BY-SA 4.0
2024-09-02T05:47:32.863Z
2024-09-09T07:50:16.553Z
2024-09-09T07:50:16.553Z
cinch
<p>I am trying to read through the proof of ELBO for diffusion models on pg. 8 of this <a href="https://arxiv.org/pdf/2208.11970" rel="nofollow noreferrer">paper</a>. However, I do not see how the author arrived at Eqn (45) from Eqn (44). Specifically, I do not know how they simplified the equation by rewriting it in terms of the KL divergence. I do know the formula for KL divergence but it does not seem like a direct application in this case. Any help is appreciated. I have included an image below of the proof for convenience.</p>
<p><a href="https://i.sstatic.net/0kE1Tm8C.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/0kE1Tm8C.jpg" alt="enter image description here" /></a></p>

CC BY-SA 4.0
2024-09-04T07:12:06.567Z
Not really an answer, but same question asked [here](https://stats.stackexchange.com/q/653800/429415)
randomforest42
<deep-learning><variational-autoencoder><diffusion-models><evidence-lower-bound><kl-divergence>
Deriving ELBO for Diffusion Models
Nikhil Sridhar
# Deriving ELBO for Diffusion Models

(Asked by: username_0 on 2024-08-23)

I am trying to read through the proof of ELBO for diffusion models on pg. 8 of this [paper](https://arxiv.org/pdf/2208.11970). However, I do not see how the author arrived at Eqn (45) from Eqn (44). Specifically, I do not know how they simplified the equation by rewriting it in terms of the KL divergence. I do know the formula for KL divergence but it does not seem like a direct application in this case. Any help is appreciated. I have included an image below of the proof for convenience.

[![enter image description here](https://i.sstatic.net/0kE1Tm8C.jpg)](https://i.sstatic.net/0kE1Tm8C.jpg)

**Tags:** deep-learning, variational-autoencoder, diffusion-models, evidence-lower-bound, kl-divergence

> **Comments:**
> **username_1**: Not really an answer, but same question asked [here](https://stats.stackexchange.com/q/653800/429415)  

---

## 1 answers

### Answer by username_2 on 2024-09-02. Score: 0 | Accepted answer

Indeed the notations here are a little involved and confusing since there exist multiple joint conditional random variables here. First please note here all involved conditional random variables are *all* conditioned only on the fixed value of the input data $x\_0$, so we can tentatively and effectively ignore it when considering other (conditional) random variables. Next let's look at the second term in your asked line (44): $$\mathbb{E}\_{q(x\_{T−1}, x\_T | x\_0)} [\log \frac{p(x\_T)}{q(x\_T | x\_{T −1})}]$$

Notice compared to the usual KL divergence, there's an additional fixed conditioning value of $x\_{T−1}$ as given data involved in the posterior $q$-distribution of the random variable $x\_T$ in above term. And since the random variable $x\_{T−1}$ is taken *jointly* with the random variable $x\_T$ given $x\_0$ over the expectation in above term, we have to take expectation wrt the random variable $x\_{T−1}$ *after* rewriting the above term as KL divergence. With these ideas we can expand the above term as:
$$\mathbb{E}\_{x\_{T−1}|x\_0 \sim q(x\_{T−1}| x\_0)}[\mathbb{E}\_{x\_T|x\_{T-1}, x\_0 \sim q(x\_T|x\_{T-1}, x\_0)} [\log \frac{p(x\_T)}{q(x\_T | x\_{T −1})}]] \\= -\mathbb{E}\_{q(x\_{T−1}| x\_0)}[\mathbb{E}\_{q(x\_T|x\_{T-1})} [\log \frac{q(x\_T | x\_{T −1})}{p(x\_T)}]] \\= -\mathbb{E}\_{q(x\_{T−1}| x\_0)}[D\_{KL}(q(x\_T | x\_{T −1})||p(x\_T))]$$

Similar process can be applied to the third term to explain the additional expectation operator of the rewritten KL divergence.


---

