Answers.Body: <p>I don't want to think about the correctness of your supposed ELBO equation now. Nevertheless, it's true that the ELBO can be rewritten in different ways (e.g. if you expand the KL divergence below, by applying its definition, you will end up with a different but equivalent version of the ELBO). I will use the most common (and definitely most intuitive, at least to me) one, which you can find in <a href="https://arxiv.org/pdf/1312.6114.pdf" rel="nofollow noreferrer">the paper that originally introduced the VAE</a>, which you should use as a reference, when you are confused (although that paper may require at least 2 readings before you fully understand it).</p>
  <p>Here's the most common form of the ELBO (for the VAE), which immediately explains the common VAE implementations (including yours):</p>
  <p><span class="math-container">$$\mathcal{L}\left(\boldsymbol{\theta}, \boldsymbol{\phi} ; \mathbf{x}^{(i)}\right)=
  \underbrace{
  \color{red}{
  -D_{K L}\left(q_{\boldsymbol{\phi}}\left(\mathbf{z} \mid \mathbf{x}^{(i)}\right) \| p_{\boldsymbol{\theta}}(\mathbf{z})\right)
  }
  }_{\text{KL divergence}}
  +
  \underbrace{
  \color{green}{
  \mathbb{E}_{q_{\boldsymbol{\phi}}\left(\mathbf{z} \mid \mathbf{x}^{(i)}\right)}\left[\color{blue}{ \log p_{\boldsymbol{\theta}}\left(\mathbf{x}^{(i)} \mid \mathbf{z}\right)}\right]
  }
  }_{\text{Expected log-likelihood}},
  $$</span></p>
  <p>where</p>
  <ul>
  <li><p><span class="math-container">$p_{\boldsymbol{\theta}}(\mathbf{z})$</span> is the prior over <span class="math-container">$\mathbf{z}$</span> (i.e. the latent variable); <strong>in practice</strong>, the prior is not actually parametrized by <span class="math-container">$\boldsymbol{\theta}$</span> (i.e. it's just a Gaussian with mean zero and variance one, or whatever, depending on your assumptions about <span class="math-container">$\mathbf{z}$</span>!), but in the paper they assume that <span class="math-container">$\mathbf{z}$</span> depends on <span class="math-container">$\boldsymbol{\theta}$</span> (see figure 1).</p>
  </li>
  <li><p><span class="math-container">$q_{\boldsymbol{\phi}}\left(\mathbf{z} \mid \mathbf{x}^{(i)}\right)$</span> is the encoder parametrized by <span class="math-container">$\boldsymbol{\phi}$</span></p>
  </li>
  <li><p><span class="math-container">$p_{\boldsymbol{\theta}}$</span> is the decoder, parametrized by <span class="math-container">$\boldsymbol{\theta}$</span></p>
  </li>
  </ul>
  <p>If you assume that <span class="math-container">$p_{\boldsymbol{\theta}}(\mathbf{z})$</span> and <span class="math-container">$q_{\boldsymbol{\phi}}\left(\mathbf{z} \mid \mathbf{x}^{(i)}\right) $</span> are Gaussian distributions, then it turns out that the KL divergence has an analytical form, which is also derived in appendix B of the VAE paper (<a href="https://ai.stackexchange.com/a/26408/2444">here</a> is a detailed derivation)</p>
  <p><span class="math-container">$$
  \color{red}{
  -D_{K L}\left(q_{\boldsymbol{\phi}}\left(\mathbf{z} \mid \mathbf{x}^{(i)}\right) \| p_{\boldsymbol{\theta}}(\mathbf{z})\right)}
  = 
  \color{red}{
  \frac{1}{2} \sum_{j=1}^{J}\left(1+\log \left(\left(\sigma_{j}\right)^{2}\right)-\left(\mu_{j}\right)^{2}-\left(\sigma_{j}\right)^{2}\right)
  }
  $$</span></p>
  <p>Hence <a href="https://github.com/pytorch/examples/blob/master/vae/main.py#L80" rel="nofollow noreferrer">this implementation</a>, which should be equivalent to your term <span class="math-container">$0.5*(1+Z_{\sigma}-Z_{\mu}^2-exp(Z_\sigma))$</span></p>
  <p>The expected log-likelihood is actually an expectation, so you cannot compute it exactly, in general. Hence, you can approximate it with Monte Carlo sampling (aka sampling averages: remember the law of large numbers?). More concretely, if you assume that you have a Bernoulli likelihood, i.e. <span class="math-container">$p_{\boldsymbol{\theta}}\left(\mathbf{x}^{(i)} \mid \mathbf{z}\right)$</span> is a Bernoulli, then its definition is (again from the VAE paper, Appendix C.1)</p>
  <p><span class="math-container">$$
  \color{blue}{ \log p(\mathbf{x} \mid \mathbf{z})}= \color{blue}{ \sum_{i=1}^{D} x_{i} \log y_{i}+\left(1-x_{i}\right) \cdot \log \left(1-y_{i}\right) }\tag{1}\label{1},
  $$</span>
  where <span class="math-container">$\mathbf{y}$</span> is the output of the decoder (i.e. the reconstruction/generation of the original input).</p>
  <p>This formula should be very familiar to you if you are familiar with the cross-entropy. In fact, minimizing the cross-entropy is equivalent to maximizing the log-likelihood (this may still be confusing because of the flipped signs in the ELBO above, but just remember that maximizing a function is equivalent to minimizing its negative!). Hence <a href="https://github.com/pytorch/examples/blob/master/vae/main.py#L73" rel="nofollow noreferrer">this loss</a>.</p>
  <p>To answer/address some of your questions/doubts directly</p>
  <blockquote>
  <p>how <span class="math-container">$(0.5*(1+Z_{\sigma}-Z_{\mu}^2-exp(Z_\sigma)))$</span> equates to the entropy?</p>
  </blockquote>
  <p>I answered this above. That's just the analytical expression for the KL divergence (by the way, the KL divergence is also known as <em>relative entropy</em>). See <a href="https://ai.stackexchange.com/a/26408/2444">this answer</a> for a derivation.</p>
  <blockquote>
  <p>It still seems a bit off because you're getting the cross entropy of <span class="math-container">$Y$</span> with it's reconstruction, not the Gaussian distribution for learning latents <span class="math-container">$Z$</span>.</p>
  </blockquote>
  <p>As you can see from the definition of the ELBO above (from the VAE paper), the expected log-likelihood is, as the name suggests, an expectation, with respect to the encoder (i.e. the Gaussian, in case you choose a Gaussian). However, the equivalence is between the log-likelihood (which is the term <strong>inside</strong> the expectation) and the cross-entropy, i.e. once you have sampled from the encoder, you just need to compute the term inside the expectation (i.e. the cross-entropy). Your term <span class="math-container">$\text{CrossEntropy}(\hat{Y}, Y)$</span> represents the CE but <strong>after you have sampled a latent variable from the encoder (or Gaussian)</strong>, otherwise, you could not have obtained the reconstruction <span class="math-container">$\hat{Y}$</span> (i.e. the reconstruction depends on this latent variable, see figure 1).</p>
  <p>In equation \ref{1}, note that there is no expectation. In fact, in the implementations, you may just sample <strong>once</strong> from the encoder, and then immediately compute the ELBO. I have seen this also in the implementations of Bayesian neural networks (basically, normal neural networks with the same principles of the VAE). However, in principle, you could sample multiple times from the Gaussian encoder, compute \ref{1} multiple times, then average it, to compute a better approximation of the <strong>expected</strong> log-likelihood.</p>
  <p>Hopefully, some of the information in this answer is clear. Honestly, I don't have much time to write a better answer now (maybe I will come back later). In any case, I think you can find all answers to your questions/doubts in the VAE paper (although, as I said, you may need to read it at least twice to understand it).</p>
  <p>By the way, the simplest/cleanest implementation of the VAE that I have found so far is <a href="https://github.com/pytorch/examples/blob/master/vae/main.py" rel="nofollow noreferrer">this one</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-12T23:25:50.587Z
Answers.Text: Really good explanation, every part of my question is answered except maybe the $CE(\hat{Y}, Y)$ being equivalent to expected log likelihood.
Answers.UserDisplayName: user8714896
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-12T23:28:09.960Z
Answers.Text: @user8714896 I think you can find the answer to that question on the web. If not, you can ask another separate question, and maybe later I will answer.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-12T22:15:09.657Z
Answers.LastActivityDate: 2022-06-05T09:10:46.570Z
Answers.LastEditDate: 2022-06-05T09:10:46.570Z
Answers.UserDisplayName: nbro
Body: <p>For a lot of VAE implementations I've seen in code, it's not really obvious to me how it equates to ELBO.</p>
  <p><span class="math-container">$$L(X)=H(Q)-H(Q:P(X,Z))=\sum_ZQ(Z)logP(Z,X)-\sum_ZQ(Z)log(Q(Z))$$</span></p>
  <p>The above is the definition of ELBO, where <span class="math-container">$X$</span> is some input, <span class="math-container">$Z$</span> is a latent variable, <span class="math-container">$H()$</span> is the entropy. <span class="math-container">$Q()$</span> is a distribution being used to approximate distribution <span class="math-container">$P()$</span>, which in the above case both <span class="math-container">$P()$</span> and <span class="math-container">$Q()$</span> are discrete distributions, because of the sum.</p>
  <p>A lot of the times when VAEs are built for reconstructing discrete data types, let's say for example an image, where each pixel can be black or white or <span class="math-container">$0$</span> or <span class="math-container">$1$</span>. The main steps of a VAE that I've seen in code are as follows:</p>
  <ol>
  <li><span class="math-container">$\text{Encoder}(Y) \rightarrow Z_u, Z_{\sigma}$</span></li>
  <li><span class="math-container">$\text{Reparameterization Trick}(Z_\mu, Z_\sigma) \rightarrow Z$</span></li>
  <li><span class="math-container">$\text{Decoder}(Z) \rightarrow \hat{Y}$</span></li>
  <li><span class="math-container">$L(Y)= \text{CrossEntropy}(\hat{Y}, Y) - 0.5*(1+Z_{\sigma}-Z_{\mu}^2-exp(Z_\sigma))$</span></li>
  </ol>
  <p>where</p>
  <ul>
  <li><span class="math-container">$Z$</span> represents the latent embedding of the auto-encoder</li>
  <li><span class="math-container">$Z_\mu$</span> and <span class="math-container">$Z_\sigma$</span> represent the mean and standard deviation for sampling for <span class="math-container">$Z$</span> from a Gaussian distribution.</li>
  <li><span class="math-container">$Y$</span> represents the binary image trying to be reconstructed</li>
  <li><span class="math-container">$\hat{Y}$</span> represents its reconstruction from the VAE.</li>
  </ul>
  <p>As we can see from the ELBO, it's the entropy of the latent distribution being learned, <span class="math-container">$Q()$</span>, which is a Gaussian, and the cross entropy of the latent distribution being learned <span class="math-container">$Q()$</span> and the actual distribution <span class="math-container">$P()$</span> with <span class="math-container">$Z$</span> intersected with <span class="math-container">$X$</span>.</p>
  <p>The main points that confuse me are</p>
  <ul>
  <li>how <span class="math-container">$\text{CrossEntropy}(\hat{Y}, Y)$</span> equates to the CE of the distribution for generating latents and its Gaussian approximation, and</li>
  <li>how <span class="math-container">$(0.5*(1+Z_{\sigma}-Z_{\mu}^2-exp(Z_\sigma)))$</span> equates to the entropy</li>
  </ul>
  <p>Is it just assumed the CE of <span class="math-container">$Y$</span> with <span class="math-container">$\hat{Y}$</span> also leads to the CE of the latent distribution with it's approximation, because they're part of <span class="math-container">$\hat{Y}$</span>'s generation? It still seems a bit off because you're getting the cross entropy of <span class="math-container">$Y$</span> with it's reconstruction, not the Gaussian distribution for learning latents <span class="math-container">$Z$</span>.</p>
  <p>Note: <span class="math-container">$Z_\sigma$</span> is usually not softplused to be strictly positive as required by a Gaussian distribution, so I think that's what <span class="math-container">$exp(Z_\sigma)$</span> is for.</p>
  
Tags: <implementation><variational-autoencoder><cross-entropy><evidence-lower-bound><categorical-crossentropy>
Title: How does the implementation of the VAE's objective function equate to ELBO?
UserDisplayName: user8714896
ThreadText: # How does the implementation of the VAE's objective function equate to ELBO?
  
  (Asked by: username_0 on 2020-11-12)
  
  For a lot of VAE implementations I've seen in code, it's not really obvious to me how it equates to ELBO.
  
  $$L(X)=H(Q)-H(Q:P(X,Z))=\sum\_ZQ(Z)logP(Z,X)-\sum\_ZQ(Z)log(Q(Z))$$
  
  The above is the definition of ELBO, where $X$ is some input, $Z$ is a latent variable, $H()$ is the entropy. $Q()$ is a distribution being used to approximate distribution $P()$, which in the above case both $P()$ and $Q()$ are discrete distributions, because of the sum.
  
  A lot of the times when VAEs are built for reconstructing discrete data types, let's say for example an image, where each pixel can be black or white or $0$ or $1$. The main steps of a VAE that I've seen in code are as follows:
  
  1. $\text{Encoder}(Y) \rightarrow Z\_u, Z\_{\sigma}$
  2. $\text{Reparameterization Trick}(Z\_\mu, Z\_\sigma) \rightarrow Z$
  3. $\text{Decoder}(Z) \rightarrow \hat{Y}$
  4. $L(Y)= \text{CrossEntropy}(\hat{Y}, Y) - 0.5\*(1+Z\_{\sigma}-Z\_{\mu}^2-exp(Z\_\sigma))$
  
  where
  
  * $Z$ represents the latent embedding of the auto-encoder
  * $Z\_\mu$ and $Z\_\sigma$ represent the mean and standard deviation for sampling for $Z$ from a Gaussian distribution.
  * $Y$ represents the binary image trying to be reconstructed
  * $\hat{Y}$ represents its reconstruction from the VAE.
  
  As we can see from the ELBO, it's the entropy of the latent distribution being learned, $Q()$, which is a Gaussian, and the cross entropy of the latent distribution being learned $Q()$ and the actual distribution $P()$ with $Z$ intersected with $X$.
  
  The main points that confuse me are
  
  * how $\text{CrossEntropy}(\hat{Y}, Y)$ equates to the CE of the distribution for generating latents and its Gaussian approximation, and
  * how $(0.5\*(1+Z\_{\sigma}-Z\_{\mu}^2-exp(Z\_\sigma)))$ equates to the entropy
  
  Is it just assumed the CE of $Y$ with $\hat{Y}$ also leads to the CE of the latent distribution with it's approximation, because they're part of $\hat{Y}$'s generation? It still seems a bit off because you're getting the cross entropy of $Y$ with it's reconstruction, not the Gaussian distribution for learning latents $Z$.
  
  Note: $Z\_\sigma$ is usually not softplused to be strictly positive as required by a Gaussian distribution, so I think that's what $exp(Z\_\sigma)$ is for.
  
  **Tags:** implementation, variational-autoencoder, cross-entropy, evidence-lower-bound, categorical-crossentropy
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-11-12. Score: 6 | Accepted answer
  
  I don't want to think about the correctness of your supposed ELBO equation now. Nevertheless, it's true that the ELBO can be rewritten in different ways (e.g. if you expand the KL divergence below, by applying its definition, you will end up with a different but equivalent version of the ELBO). I will use the most common (and definitely most intuitive, at least to me) one, which you can find in [the paper that originally introduced the VAE](https://arxiv.org/pdf/1312.6114.pdf), which you should use as a reference, when you are confused (although that paper may require at least 2 readings before you fully understand it).
  
  Here's the most common form of the ELBO (for the VAE), which immediately explains the common VAE implementations (including yours):
  
  $$\mathcal{L}\left(\boldsymbol{\theta}, \boldsymbol{\phi} ; \mathbf{x}^{(i)}\right)=
  \underbrace{
  \color{red}{
  -D\_{K L}\left(q\_{\boldsymbol{\phi}}\left(\mathbf{z} \mid \mathbf{x}^{(i)}\right) \| p\_{\boldsymbol{\theta}}(\mathbf{z})\right)
  }
  }\_{\text{KL divergence}}
  +
  \underbrace{
  \color{green}{
  \mathbb{E}\_{q\_{\boldsymbol{\phi}}\left(\mathbf{z} \mid \mathbf{x}^{(i)}\right)}\left[\color{blue}{ \log p\_{\boldsymbol{\theta}}\left(\mathbf{x}^{(i)} \mid \mathbf{z}\right)}\right]
  }
  }\_{\text{Expected log-likelihood}},
  $$
  
  where
  
  * $p\_{\boldsymbol{\theta}}(\mathbf{z})$ is the prior over $\mathbf{z}$ (i.e. the latent variable); **in practice**, the prior is not actually parametrized by $\boldsymbol{\theta}$ (i.e. it's just a Gaussian with mean zero and variance one, or whatever, depending on your assumptions about $\mathbf{z}$!), but in the paper they assume that $\mathbf{z}$ depends on $\boldsymbol{\theta}$ (see figure 1).
  * $q\_{\boldsymbol{\phi}}\left(\mathbf{z} \mid \mathbf{x}^{(i)}\right)$ is the encoder parametrized by $\boldsymbol{\phi}$
  * $p\_{\boldsymbol{\theta}}$ is the decoder, parametrized by $\boldsymbol{\theta}$
  
  If you assume that $p\_{\boldsymbol{\theta}}(\mathbf{z})$ and $q\_{\boldsymbol{\phi}}\left(\mathbf{z} \mid \mathbf{x}^{(i)}\right) $ are Gaussian distributions, then it turns out that the KL divergence has an analytical form, which is also derived in appendix B of the VAE paper ([here](https://ai.stackexchange.com/a/26408/2444) is a detailed derivation)
  
  $$
  \color{red}{
  -D\_{K L}\left(q\_{\boldsymbol{\phi}}\left(\mathbf{z} \mid \mathbf{x}^{(i)}\right) \| p\_{\boldsymbol{\theta}}(\mathbf{z})\right)}
  =
  \color{red}{
  \frac{1}{2} \sum\_{j=1}^{J}\left(1+\log \left(\left(\sigma\_{j}\right)^{2}\right)-\left(\mu\_{j}\right)^{2}-\left(\sigma\_{j}\right)^{2}\right)
  }
  $$
  
  Hence [this implementation](https://github.com/pytorch/examples/blob/master/vae/main.py#L80), which should be equivalent to your term $0.5\*(1+Z\_{\sigma}-Z\_{\mu}^2-exp(Z\_\sigma))$
  
  The expected log-likelihood is actually an expectation, so you cannot compute it exactly, in general. Hence, you can approximate it with Monte Carlo sampling (aka sampling averages: remember the law of large numbers?). More concretely, if you assume that you have a Bernoulli likelihood, i.e. $p\_{\boldsymbol{\theta}}\left(\mathbf{x}^{(i)} \mid \mathbf{z}\right)$ is a Bernoulli, then its definition is (again from the VAE paper, Appendix C.1)
  
  $$
  \color{blue}{ \log p(\mathbf{x} \mid \mathbf{z})}= \color{blue}{ \sum\_{i=1}^{D} x\_{i} \log y\_{i}+\left(1-x\_{i}\right) \cdot \log \left(1-y\_{i}\right) }\tag{1}\label{1},
  $$
  where $\mathbf{y}$ is the output of the decoder (i.e. the reconstruction/generation of the original input).
  
  This formula should be very familiar to you if you are familiar with the cross-entropy. In fact, minimizing the cross-entropy is equivalent to maximizing the log-likelihood (this may still be confusing because of the flipped signs in the ELBO above, but just remember that maximizing a function is equivalent to minimizing its negative!). Hence [this loss](https://github.com/pytorch/examples/blob/master/vae/main.py#L73).
  
  To answer/address some of your questions/doubts directly
  
  > how $(0.5\*(1+Z\_{\sigma}-Z\_{\mu}^2-exp(Z\_\sigma)))$ equates to the entropy?
  
  I answered this above. That's just the analytical expression for the KL divergence (by the way, the KL divergence is also known as *relative entropy*). See [this answer](https://ai.stackexchange.com/a/26408/2444) for a derivation.
  
  > It still seems a bit off because you're getting the cross entropy of $Y$ with it's reconstruction, not the Gaussian distribution for learning latents $Z$.
  
  As you can see from the definition of the ELBO above (from the VAE paper), the expected log-likelihood is, as the name suggests, an expectation, with respect to the encoder (i.e. the Gaussian, in case you choose a Gaussian). However, the equivalence is between the log-likelihood (which is the term **inside** the expectation) and the cross-entropy, i.e. once you have sampled from the encoder, you just need to compute the term inside the expectation (i.e. the cross-entropy). Your term $\text{CrossEntropy}(\hat{Y}, Y)$ represents the CE but **after you have sampled a latent variable from the encoder (or Gaussian)**, otherwise, you could not have obtained the reconstruction $\hat{Y}$ (i.e. the reconstruction depends on this latent variable, see figure 1).
  
  In equation \ref{1}, note that there is no expectation. In fact, in the implementations, you may just sample **once** from the encoder, and then immediately compute the ELBO. I have seen this also in the implementations of Bayesian neural networks (basically, normal neural networks with the same principles of the VAE). However, in principle, you could sample multiple times from the Gaussian encoder, compute \ref{1} multiple times, then average it, to compute a better approximation of the **expected** log-likelihood.
  
  Hopefully, some of the information in this answer is clear. Honestly, I don't have much time to write a better answer now (maybe I will come back later). In any case, I think you can find all answers to your questions/doubts in the VAE paper (although, as I said, you may need to read it at least twice to understand it).
  
  By the way, the simplest/cleanest implementation of the VAE that I have found so far is [this one](https://github.com/pytorch/examples/blob/master/vae/main.py).
  
  > **Comments:**
  > **username_0**: Really good explanation, every part of my question is answered except maybe the $CE(\hat{Y}, Y)$ being equivalent to expected log likelihood.  
  > **username_1**: @username_0 I think you can find the answer to that question on the web. If not, you can ask another separate question, and maybe later I will answer.  
  
  ---
  
  