Answers.Body: <p>The VAE uses the <strong>ELBO loss</strong>, which is composed of the KL term and the likelihood term. The ELBO loss is a lower bound on the evidence of <strong>your</strong> data, so if you maximize the ELBO you also maximize the evidence of the given data, which is what you indirectly want to do, i.e. you want the probability of your given data (i.e. the data in your dataset) to be high (because you want to use the VAE for the generation of inputs similar to the ones in your dataset). So, the idea is that you optimize both the KL term and the reconstruction (or likelihood) term <strong>jointly</strong> (i.e. the ELBO). Why? Because, as I just said, the ELBO is the <strong>E</strong>vidence <strong>L</strong>ower <strong>BO</strong>und on the given data, so, by maximizing it, you are also maximizing the evidence of your data. In other words, if you maximize the ELBO, you are finding a decoder that will have a high probability of reconstructing your inputs (i.e. the likelihood term), but, at the same time, you want your encoder to be constrained (i.e. KL term). Please, read <a href="https://ai.stackexchange.com/a/17900/2444">this answer</a> for further details.</p>
  
  <blockquote>
    <p>Here, I cannot understand why the sampled <span class="math-container">$z$</span> should make the original image, since the <span class="math-container">$z$</span> is sampled, it seems that the <span class="math-container">$z$</span> does not have any relationship between the original image.</p>
  </blockquote>
  
  <p>The relationship is that you will be maximizing the ELBO, which implies (and you can see this implication only if you are familiar with the ELBO loss) you will be minimizing the KL divergence between your posterior and the prior to generate the samples <span class="math-container">$z$</span> (i.e. minimizing because there will be a minus in front of the KL term in the ELBO loss) and maximizing the probability of the reconstructed input. More precisely, <span class="math-container">$z$</span> is used to reconstruct the input (i.e. the decoder does this), which is then used to calculate the reconstruction loss. </p>
  
  <p>In the mathematical formulations, you will see that the likelihood term of the ELBO is <span class="math-container">$p(x \mid z)$</span>, i.e. the likelihood of the input <span class="math-container">$x$</span> given <span class="math-container">$z$</span>. The <span class="math-container">$z$</span> is the input to the decoder, which produces a reconstruction of <span class="math-container">$x$</span>. In practice, people will e.g. use the cross-entropy to then calculate the "reconstruction loss" (e.g. see <a href="https://github.com/pytorch/examples/blob/master/vae/main.py" rel="nofollow noreferrer">this PyTorch implementation</a>), which <em>should</em> correspond to this likelihood term <span class="math-container">$p(x \mid z)$</span>. Why does the cross-entropy correspond to a likelihood? Because you can actually prove that the cross-entropy is equivalent to the negative log-likelihood. (Also, note that, in the ELBO loss, <span class="math-container">$p(x \mid z)$</span> does not appear, but the logarithm of <span class="math-container">$p(x \mid z)$</span> appears, but, for simplicity, I have used <span class="math-container">$p(x \mid z)$</span> rather than <span class="math-container">$\log p(x \mid z)$</span> above.)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-26T13:28:55.190Z
Answers.Text: This answer could potentially make you more confused. The concepts aren't really straightforward and it takes several iterations until you get them, so, please, try to read my answer and other sources related to the topic several times.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-26T13:13:51.763Z
Answers.LastActivityDate: 2020-03-26T20:48:04.810Z
Answers.LastEditDate: 2020-03-26T20:48:04.810Z
Answers.UserDisplayName: nbro
Body: <p>VAE is trained to reduce the following two losses.</p>
  <ol>
  <li><p>KL divergence between inferred latent distribution and Gaussian.</p>
  </li>
  <li><p>the reconstruction loss</p>
  </li>
  </ol>
  <p>I understand that the first one regularizes VAE to get structured latent space. But why and how does the second loss help VAE to work?</p>
  <p>During the training of the VAE, we first feed an image to the encoder. Then, the encoder infers mean and variance. After that, we sample <span class="math-container">$z$</span> from the inferred distribution. Finally, the decoder gets the sampled <span class="math-container">$z$</span> and generates an image. So, in this way, the VAE is trained to make the generated image to be equal to the original input image.</p>
  <p>Here, I cannot understand why the sampled <span class="math-container">$z$</span> should make the original image, since the <span class="math-container">$z$</span> is sampled, it seems that the <span class="math-container">$z$</span> does not have any relationship between the original image.</p>
  <p>But, as you know, VAE works well. So I think I miss something important or understand it in a totally wrong way.</p>
  
Tags: <deep-learning><math><variational-autoencoder><evidence-lower-bound>
Title: Why does the variational auto-encoder use the reconstruction loss?
UserDisplayName: Jun
ThreadText: # Why does the variational auto-encoder use the reconstruction loss?
  
  (Asked by: username_0 on 2020-03-26)
  
  VAE is trained to reduce the following two losses.
  
  1. KL divergence between inferred latent distribution and Gaussian.
  2. the reconstruction loss
  
  I understand that the first one regularizes VAE to get structured latent space. But why and how does the second loss help VAE to work?
  
  During the training of the VAE, we first feed an image to the encoder. Then, the encoder infers mean and variance. After that, we sample $z$ from the inferred distribution. Finally, the decoder gets the sampled $z$ and generates an image. So, in this way, the VAE is trained to make the generated image to be equal to the original input image.
  
  Here, I cannot understand why the sampled $z$ should make the original image, since the $z$ is sampled, it seems that the $z$ does not have any relationship between the original image.
  
  But, as you know, VAE works well. So I think I miss something important or understand it in a totally wrong way.
  
  **Tags:** deep-learning, math, variational-autoencoder, evidence-lower-bound
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-03-26. Score: 2 | Accepted answer
  
  The VAE uses the **ELBO loss**, which is composed of the KL term and the likelihood term. The ELBO loss is a lower bound on the evidence of **your** data, so if you maximize the ELBO you also maximize the evidence of the given data, which is what you indirectly want to do, i.e. you want the probability of your given data (i.e. the data in your dataset) to be high (because you want to use the VAE for the generation of inputs similar to the ones in your dataset). So, the idea is that you optimize both the KL term and the reconstruction (or likelihood) term **jointly** (i.e. the ELBO). Why? Because, as I just said, the ELBO is the **E**vidence **L**ower **BO**und on the given data, so, by maximizing it, you are also maximizing the evidence of your data. In other words, if you maximize the ELBO, you are finding a decoder that will have a high probability of reconstructing your inputs (i.e. the likelihood term), but, at the same time, you want your encoder to be constrained (i.e. KL term). Please, read [this answer](https://ai.stackexchange.com/a/17900/2444) for further details.
  
  > Here, I cannot understand why the sampled $z$ should make the original image, since the $z$ is sampled, it seems that the $z$ does not have any relationship between the original image.
  
  The relationship is that you will be maximizing the ELBO, which implies (and you can see this implication only if you are familiar with the ELBO loss) you will be minimizing the KL divergence between your posterior and the prior to generate the samples $z$ (i.e. minimizing because there will be a minus in front of the KL term in the ELBO loss) and maximizing the probability of the reconstructed input. More precisely, $z$ is used to reconstruct the input (i.e. the decoder does this), which is then used to calculate the reconstruction loss.
  
  In the mathematical formulations, you will see that the likelihood term of the ELBO is $p(x \mid z)$, i.e. the likelihood of the input $x$ given $z$. The $z$ is the input to the decoder, which produces a reconstruction of $x$. In practice, people will e.g. use the cross-entropy to then calculate the "reconstruction loss" (e.g. see [this PyTorch implementation](https://github.com/pytorch/examples/blob/master/vae/main.py)), which *should* correspond to this likelihood term $p(x \mid z)$. Why does the cross-entropy correspond to a likelihood? Because you can actually prove that the cross-entropy is equivalent to the negative log-likelihood. (Also, note that, in the ELBO loss, $p(x \mid z)$ does not appear, but the logarithm of $p(x \mid z)$ appears, but, for simplicity, I have used $p(x \mid z)$ rather than $\log p(x \mid z)$ above.)
  
  > **Comments:**
  > **username_1**: This answer could potentially make you more confused. The concepts aren't really straightforward and it takes several iterations until you get them, so, please, try to read my answer and other sources related to the topic several times.  
  
  ---
  
  