Answers.Body: <p>Ans 1.</p>
  <p>The motive of Variational Inference(on which VAE is based), is to decrease <span class="math-container">$KL(q(z|x)||p(z))$</span>, where p(z) is our chosen distribution of the hidden variable z. After doing some math, we can write this expression as-</p>
  <p><span class="math-container">$
  KL(q||x) = log(p(x)) - \Sigma_z q(z)log(\frac{p(x,z)}{q(z)})
  $</span></p>
  <p>For a given x, the first term of RHS is constant. So we maximise the second term so that the KL divergence goes to zero.</p>
  <p>We can write the second term as</p>
  <p><span class="math-container">$E_{q(z)}[log(p(x|z)] - KL(q(z|x)||p(z))$</span></p>
  <p>(try writing p(x,z) as <span class="math-container">$\frac{p(x|z)}{p(z)}$</span> and then expand. Here, p(z) is the distribution of our choice, i.e. a Gaussian distribution). We argue that the process from z to x is deterministic and z is gaussian. So, the first term could be replaced by <span class="math-container">$exp(log(-||x-\hat{x}||^2))$</span>(we replaced z by <span class="math-container">$\hat{x}$</span> because it's deterministic- this is now the exact proof). So we have-</p>
  <p><span class="math-container">$
  maximize(-||x-\hat(x)||-KL(q(z|x)||p(z)))
  $</span></p>
  <p>and we get our loss function.</p>
  <p>We also know that variational autoencoders <strong>almost never</strong> find the optimal solution, so I am not sure how would playing around with the weights affect it(Nor do I know if it makes sense mathematically).</p>
  <p>Ans 2.</p>
  <p>We can say that the KL divergence has a regularising effect.</p>
  <p><a href="https://towardsdatascience.com/intuitively-understanding-variational-autoencoders-1bfe67eb5daf" rel="nofollow noreferrer">This page</a> has some nice experiments which will help you understand what happens to the latent space when you decrease the KL divergence part.</p>
  <p>Ans 3.</p>
  <p>Yes, you can say that. You are fixing the dimensions, but are lenient on the distribution. In fact, you are approaching autoencoders by doing this.</p>
  <p>Separate-</p>
  <p>I want to point you towards <a href="https://ermongroup.github.io/cs228-notes/inference/variational/" rel="nofollow noreferrer">this</a> article. It explains why we choose to minimise <span class="math-container">$KL(q(z|x)||p(z))$</span> instead of <span class="math-container">$KL(p(z)||q(z|x))$</span> (the latter is intractable) and what would happen if we choose less independent variables for our estimator q(z).</p>
  <p>Also, have you tried increasing the dimensions of the latent space? It can also have a 'de-regularizing' effect. It seems that the model is underfitting the data- the reconstruction loss is high with the normal loss, compared to when you decrease the regularizing term.</p>
  <p>Hope it helps.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-06T01:51:30.553Z
Answers.Text: I have to read this answer carefully to make sure that all info is correct. However, I would like to point out that there's some discussion (in the literature, so you can find some papers that talk about it) on how to scale the KL divergence term in the loss functions of Bayesian neural networks (based on variational inference, i.e. mean-field variational Bayesian neural networks), which have a loss function similar to the VAE, i.e. they also have the KL divergence term. So, you may want to have a look at those papers and maybe edit your answer to include some info that you find there.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-05T17:28:16.917Z
Answers.LastActivityDate: 2020-11-05T17:55:11.660Z
Answers.LastEditDate: 2020-11-05T17:55:11.660Z
Answers.UserDisplayName: cybershiptrooper
Body: <p>Variational autoencoders have two components in their loss function. The first component is the reconstruction loss, which for image data, is the pixel-wise difference between the input image and output image. The second component is the Kullback–Leibler divergence which is introduced in order to make image encodings in the latent space more 'smooth'. Here is the loss function:</p>
  <p><span class="math-container">\begin{align}
  \text { loss }
  &amp;=
  \|x-\hat{x}\|^{2}+\operatorname{KL}\left[N\left(\mu_{x}, \sigma_{x}\right), \mathrm{N}(0,1)\right] \\
  &amp;=
  \|x-\mathrm{d}(z)\|^{2}+\operatorname{KL}\left[N\left(\mu_{x^{\prime}} \sigma_{x}\right), \mathrm{N}(0,1)\right]
  \end{align}</span></p>
  <p>I am running some experiments on a dataset of famous artworks using Variational Autoencoders. My question concerns scaling the two components of the loss function in order to manipulate the training procedure to achieve better results.</p>
  <p>I present two scenarios. The first scenario does not scale the loss components.
  <a href="https://i.sstatic.net/zOs3u.png" rel="noreferrer"><img src="https://i.sstatic.net/zOs3u.png" alt="VAE - no scaling" /></a></p>
  <p>Here you can see the two components of the loss function. Observe that the order of magnitude of the Kullback–Leibler divergence is significantly smaller than that of the reconstruction loss. Also observe that 'my famous' paintings have become unrecognisable. The image shows the reconstructions of the input data.</p>
  <p><a href="https://i.sstatic.net/OkCvG.png" rel="noreferrer"><img src="https://i.sstatic.net/OkCvG.png" alt="unscaled reconstruction" /></a></p>
  <p>In the second scenario I have <strong>scaled the KL term with 0.1</strong>. Now we can see that the reconstructions are looking much better.</p>
  <p><a href="https://i.sstatic.net/Xc9zP.png" rel="noreferrer"><img src="https://i.sstatic.net/Xc9zP.png" alt="VAE - with scaling" /></a>
  <a href="https://i.sstatic.net/N9Fvi.png" rel="noreferrer"><img src="https://i.sstatic.net/N9Fvi.png" alt="Scaled Reconstruction" /></a></p>
  <p><strong>Question</strong></p>
  <ol>
  <li><p>Is it mathematically sound to train the network by scaling the components of the loss function? Or am I effectively excluding the KL term in the optimisation?</p>
  </li>
  <li><p>How to understand this in terms of gradient descent?</p>
  </li>
  <li><p>Is it fair to say that we are telling the model &quot;we care more about the image reconstructions than 'smoothing' the latent space&quot;?</p>
  </li>
  </ol>
  <p>I am confident that my network design (convolutional layers, latent vector size) have the capacity to learn parameters to create proper reconstructions as a Convolutional Autoencoder with the same parameters is able to reconstruct perfectly.</p>
  <p><a href="https://ai.stackexchange.com/q/5716/2444">Here</a> is a similar question.</p>
  <p>Image Reference:
  <a href="https://towardsdatascience.com/understanding-variational-autoencoders-vaes-f70510919f73" rel="noreferrer">https://towardsdatascience.com/understanding-variational-autoencoders-vaes-f70510919f73</a></p>
  
Tags: <objective-functions><gradient-descent><variational-autoencoder><kl-divergence>
Title: What is the impact of scaling the KL divergence and reconstruction loss in the VAE objective function?
UserDisplayName: rich
ThreadText: # What is the impact of scaling the KL divergence and reconstruction loss in the VAE objective function?
  
  (Asked by: username_0 on 2020-11-05)
  
  Variational autoencoders have two components in their loss function. The first component is the reconstruction loss, which for image data, is the pixel-wise difference between the input image and output image. The second component is the Kullback–Leibler divergence which is introduced in order to make image encodings in the latent space more 'smooth'. Here is the loss function:
  
  \begin{align}
  \text { loss }
  &=
  \|x-\hat{x}\|^{2}+\operatorname{KL}\left[N\left(\mu\_{x}, \sigma\_{x}\right), \mathrm{N}(0,1)\right] \\
  &=
  \|x-\mathrm{d}(z)\|^{2}+\operatorname{KL}\left[N\left(\mu\_{x^{\prime}} \sigma\_{x}\right), \mathrm{N}(0,1)\right]
  \end{align}
  
  I am running some experiments on a dataset of famous artworks using Variational Autoencoders. My question concerns scaling the two components of the loss function in order to manipulate the training procedure to achieve better results.
  
  I present two scenarios. The first scenario does not scale the loss components.
  [![VAE - no scaling](https://i.sstatic.net/zOs3u.png)](https://i.sstatic.net/zOs3u.png)
  
  Here you can see the two components of the loss function. Observe that the order of magnitude of the Kullback–Leibler divergence is significantly smaller than that of the reconstruction loss. Also observe that 'my famous' paintings have become unrecognisable. The image shows the reconstructions of the input data.
  
  [![unscaled reconstruction](https://i.sstatic.net/OkCvG.png)](https://i.sstatic.net/OkCvG.png)
  
  In the second scenario I have **scaled the KL term with 0.1**. Now we can see that the reconstructions are looking much better.
  
  [![VAE - with scaling](https://i.sstatic.net/Xc9zP.png)](https://i.sstatic.net/Xc9zP.png)
  [![Scaled Reconstruction](https://i.sstatic.net/N9Fvi.png)](https://i.sstatic.net/N9Fvi.png)
  
  **Question**
  
  1. Is it mathematically sound to train the network by scaling the components of the loss function? Or am I effectively excluding the KL term in the optimisation?
  2. How to understand this in terms of gradient descent?
  3. Is it fair to say that we are telling the model "we care more about the image reconstructions than 'smoothing' the latent space"?
  
  I am confident that my network design (convolutional layers, latent vector size) have the capacity to learn parameters to create proper reconstructions as a Convolutional Autoencoder with the same parameters is able to reconstruct perfectly.
  
  [Here](https://ai.stackexchange.com/q/5716/2444) is a similar question.
  
  Image Reference:
  <https://towardsdatascience.com/understanding-variational-autoencoders-vaes-f70510919f73>
  
  **Tags:** objective-functions, gradient-descent, variational-autoencoder, kl-divergence
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-11-05. Score: 2
  
  Ans 1.
  
  The motive of Variational Inference(on which VAE is based), is to decrease $KL(q(z|x)||p(z))$, where p(z) is our chosen distribution of the hidden variable z. After doing some math, we can write this expression as-
  
  $
  KL(q||x) = log(p(x)) - \Sigma\_z q(z)log(\frac{p(x,z)}{q(z)})
  $
  
  For a given x, the first term of RHS is constant. So we maximise the second term so that the KL divergence goes to zero.
  
  We can write the second term as
  
  $E\_{q(z)}[log(p(x|z)] - KL(q(z|x)||p(z))$
  
  (try writing p(x,z) as $\frac{p(x|z)}{p(z)}$ and then expand. Here, p(z) is the distribution of our choice, i.e. a Gaussian distribution). We argue that the process from z to x is deterministic and z is gaussian. So, the first term could be replaced by $exp(log(-||x-\hat{x}||^2))$(we replaced z by $\hat{x}$ because it's deterministic- this is now the exact proof). So we have-
  
  $
  maximize(-||x-\hat(x)||-KL(q(z|x)||p(z)))
  $
  
  and we get our loss function.
  
  We also know that variational autoencoders **almost never** find the optimal solution, so I am not sure how would playing around with the weights affect it(Nor do I know if it makes sense mathematically).
  
  Ans 2.
  
  We can say that the KL divergence has a regularising effect.
  
  [This page](https://towardsdatascience.com/intuitively-understanding-variational-autoencoders-1bfe67eb5daf) has some nice experiments which will help you understand what happens to the latent space when you decrease the KL divergence part.
  
  Ans 3.
  
  Yes, you can say that. You are fixing the dimensions, but are lenient on the distribution. In fact, you are approaching autoencoders by doing this.
  
  Separate-
  
  I want to point you towards [this](https://ermongroup.github.io/cs228-notes/inference/variational/) article. It explains why we choose to minimise $KL(q(z|x)||p(z))$ instead of $KL(p(z)||q(z|x))$ (the latter is intractable) and what would happen if we choose less independent variables for our estimator q(z).
  
  Also, have you tried increasing the dimensions of the latent space? It can also have a 'de-regularizing' effect. It seems that the model is underfitting the data- the reconstruction loss is high with the normal loss, compared to when you decrease the regularizing term.
  
  Hope it helps.
  
  > **Comments:**
  > **username_2**: I have to read this answer carefully to make sure that all info is correct. However, I would like to point out that there's some discussion (in the literature, so you can find some papers that talk about it) on how to scale the KL divergence term in the loss functions of Bayesian neural networks (based on variational inference, i.e. mean-field variational Bayesian neural networks), which have a loss function similar to the VAE, i.e. they also have the KL divergence term. So, you may want to have a look at those papers and maybe edit your answer to include some info that you find there.  
  
  ---
  
  