Answers.Body: <p>On page 5 of <a href="https://arxiv.org/pdf/1312.6114.pdf#page=5" rel="nofollow noreferrer">the VAE paper</a>, it's clearly stated</p>
  <blockquote>
  <p>We let <span class="math-container">$p_{\boldsymbol{\theta}}(\mathbf{x} \mid \mathbf{z})$</span> be a multivariate Gaussian (in case of real-valued data) or Bernoulli (in case of binary data) whose distribution parameters are computed from <span class="math-container">$\mathbf{z}$</span> with a MLP (a fully-connected neural network with a single hidden layer, see appendix <span class="math-container">$\mathrm{C}$</span> ).</p>
  <p>...</p>
  <p>As explained above and in appendix <span class="math-container">$\mathrm{C}$</span>, the decoding term <span class="math-container">$\log p_{\boldsymbol{\theta}}\left(\mathbf{x}^{(i)} \mid \mathbf{z}^{(i, l)}\right)$</span> is a Bernoulli or Gaussian MLP, <strong>depending on the type of data we are modelling</strong>.</p>
  </blockquote>
  <p>So, if you are trying to predict real numbers (in the case of images, these can be the RGB values in the range <span class="math-container">$[0, 1]$</span>), then you can assume <span class="math-container">$p_{\boldsymbol{\theta}}(\mathbf{x} \mid \mathbf{z})$</span> is a Gaussian.</p>
  <p>It turns out that maximising the Gaussian likelihood is equivalent to minimising the MSE between the prediction of the decoder and the real image. You can easily show this: just replace <span class="math-container">$p_{\boldsymbol{\theta}}(\mathbf{x} \mid \mathbf{z})$</span> with <a href="https://en.wikipedia.org/wiki/Normal_distribution" rel="nofollow noreferrer">the Gaussian pdf</a>, then maximise that wrt the parameters, and you should end up with something that resembles the MSE. G. Hinton shows this in <a href="https://www.youtube.com/watch?v=vEPQNwxd1Y4" rel="nofollow noreferrer">this video lesson</a>. See also <a href="https://ai.stackexchange.com/a/17671/2444">this related answer</a>.</p>
  <p>So, <strong>yes, minimizing the MSE is theoretically founded</strong>, provided that you're trying to predict some real number.</p>
  <p>When the binary cross-entropy (instead of the MSE) is used (e.g. <a href="https://github.com/pytorch/examples/blob/main/vae/main.py" rel="nofollow noreferrer">here</a>), the assumption is that you're maximizing a Bernoulli likelihood (instead of a Gaussian) - this can also be easily shown.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-18T05:48:35.347Z
Answers.Text: the pdf is $\frac{1}{\sigma_z \sqrt{2\pi} } e^{-\frac{1}{2}\left(\frac{x-\mu_z}{\sigma_z}\right)^2}$. MSE is $\|\hat{x}-x\|^2$. I don't see the connection other than the fact there's an $(x-\mu_z)^2$ at the power of e. Note that in p(x|z), the original x is used, not the reconstruction.
Answers.UserDisplayName: IttayD
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-18T05:51:52.340Z
Answers.Text: See section C.2 in the original paper where they calculate p(x|z) with no MSE / BCE
Answers.UserDisplayName: IttayD
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-18T11:28:57.573Z
Answers.Text: Note sure why you say "Note that in p(x|z), the original x is used, not the reconstruction.". Both the original and the reconstructed images are used. z is used to compute the reconstructed image (or pixel).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-16T14:03:59.667Z
Answers.LastActivityDate: 2022-06-12T08:46:47.013Z
Answers.LastEditDate: 2022-06-12T08:46:47.013Z
Answers.UserDisplayName: nbro
Answers.Body: <p>If <span class="math-container">$p(x|z) \sim \mathcal{N}(f(z), I)$</span>, then</p>
  <p><span class="math-container">\begin{align}
  \log\ p(x|z) 
  &amp;\sim \log\ \exp(-(x-f(z))^2) \\
  &amp;\sim -(x-f(z))^2 \\
  &amp;= -(x-\hat{x})^2,
  \end{align}</span></p>
  <p>where <span class="math-container">$\hat{x}$</span>, the reconstructed image, is just the distribution mean <span class="math-container">$f(z)$</span>.</p>
  <p>It also makes sense to use the distribution mean when using the decoder (vs. just when training), as it is the one with the highest pdf value. So, the decoder produces a distribution from which we take the mean as our result.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-12T08:16:25.257Z
Answers.Text: This seems correct to me (and this is what I had in mind when I wrote my answer above, although I didn't write the derivation), but note that the second $\sim$ is exactly $=$. The first one is correct because you're omitting the denominator of the Gaussian density (which becomes a constant if you optimize wrt $\theta$), so you use approximately equal, which is correct. Moreover, it might be a good idea to add a underscript to $f$, i.e. $f_\theta$ to emphasize that you will be optimizing with respect to those parameters, although you jointly optimize them with $\phi$ (the encoder parameters).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-18T09:56:51.650Z
Answers.LastActivityDate: 2022-06-12T08:14:03.463Z
Answers.LastEditDate: 2022-06-12T08:14:03.463Z
Answers.UserDisplayName: IttayD
Body: <p>In VAEs, we try to maximize the ELBO = <span class="math-container">$\mathbb{E}_q [\log\ p(x|z)] + D_{KL}(q(z \mid x), p(z))$</span>, but I see that many implement the first term as the MSE of the image and its reconstruction. Here's a paper (section 5) that seems to do that: <a href="https://arxiv.org/pdf/1911.02469.pdf" rel="nofollow noreferrer">Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapse</a> (2019) by James Lucas et al. Is this mathematically sound?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-15T10:09:41.403Z
Comments.Text: It may be a good idea to provide 1-2 examples where you saw this because the MSE is not always used. For example, [here](https://github.com/pytorch/examples/blob/master/vae/main.py) they use the cross-entropy.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-15T14:06:55.337Z
Comments.Text: I recently read that MSE loss optimization is equivalent to minimizing Pearson $\chi^{2}$ divergence. Kullback–Leibler divergence (and also cross-entropy) has its [drawbacks](https://ai.stackexchange.com/a/25288/12841). Here is [explanations](https://paperswithcode.com/method/lsgan) of Least Squares loss for GAN
Comments.UserDisplayName: Aray Karjauv
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-15T14:17:13.347Z
Comments.Text: As you mentioned, MSE is used to measure the difference between the original and generated images. This encourages the model to preserve the original content. MSE loss can be used as an additional term, which is done in [CycleGAN](https://paperswithcode.com/method/cyclegan), where the authors use LSGAN loss and cycle-consistent loss, which is MSE-like loss.
Comments.UserDisplayName: Aray Karjauv
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-16T09:36:39.970Z
Comments.Text: @nbro, it is not clear why they use BCE there... In fact, that implementation doesn't seem to sample between the encoder and decoder, so even more strange. Looks like they treat the distribution parameters as the input to the decoder
Comments.UserDisplayName: IttayD
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-16T09:39:45.183Z
Comments.Text: What do you mean by "sample between the encoder and decoder"? Yes, the input to the decoder is a sample from the latent space, so I am not sure what you mean.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-16T11:05:44.820Z
Comments.Text: @nbro, I don't see in the implementation how they sample. But my main question is how BCE is achieved on data points (instead of distribution)
Comments.UserDisplayName: IttayD
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-16T11:26:29.003Z
Comments.Text: @IttayD Please, provide the link to the examples of the VAE implementations that use the MSE. To give a proper answer, I would need more context.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-18T05:51:42.263Z
Comments.Text: @nbro, MSE or BCE are the same thing. The VAE paper doesn't talk about comparing the reconstructed image with the original one. Just about optimizing PDFs. See section C.2 in the original paper where they calculate p(x|z) with no MSE / BCE
Comments.UserDisplayName: IttayD
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-18T05:54:43.123Z
Comments.Text: @nbro, from the article: "for the decoder we used MLPs with
  either Gaussian or Bernoulli output". That is, the output of the decoder are the parameters of a distribution, not a reconstructed image. We sample from it to get the image.
Comments.UserDisplayName: IttayD
Tags: <objective-functions><autoencoders><variational-autoencoder><mean-squared-error><evidence-lower-bound>
Title: In variational autoencoders, why do people use MSE for the loss?
UserDisplayName: IttayD
ThreadText: # In variational autoencoders, why do people use MSE for the loss?
  
  (Asked by: username_0 on 2021-04-15)
  
  In VAEs, we try to maximize the ELBO = $\mathbb{E}\_q [\log\ p(x|z)] + D\_{KL}(q(z \mid x), p(z))$, but I see that many implement the first term as the MSE of the image and its reconstruction. Here's a paper (section 5) that seems to do that: [Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapse](https://arxiv.org/pdf/1911.02469.pdf) (2019) by James Lucas et al. Is this mathematically sound?
  
  **Tags:** objective-functions, autoencoders, variational-autoencoder, mean-squared-error, evidence-lower-bound
  
  > **Comments:**
  > **username_1**: It may be a good idea to provide 1-2 examples where you saw this because the MSE is not always used. For example, [here](https://github.com/pytorch/examples/blob/master/vae/main.py) they use the cross-entropy.  
  > **username_2**: I recently read that MSE loss optimization is equivalent to minimizing Pearson $\chi^{2}$ divergence. Kullback–Leibler divergence (and also cross-entropy) has its [drawbacks](https://ai.stackexchange.com/a/25288/12841). Here is [explanations](https://paperswithcode.com/method/lsgan) of Least Squares loss for GAN  
  > **username_2**: As you mentioned, MSE is used to measure the difference between the original and generated images. This encourages the model to preserve the original content. MSE loss can be used as an additional term, which is done in [CycleGAN](https://paperswithcode.com/method/cyclegan), where the authors use LSGAN loss and cycle-consistent loss, which is MSE-like loss.  
  > **username_0**: @username_1, it is not clear why they use BCE there... In fact, that implementation doesn't seem to sample between the encoder and decoder, so even more strange. Looks like they treat the distribution parameters as the input to the decoder  
  > **username_1**: What do you mean by "sample between the encoder and decoder"? Yes, the input to the decoder is a sample from the latent space, so I am not sure what you mean.  
  > **username_0**: @username_1, I don't see in the implementation how they sample. But my main question is how BCE is achieved on data points (instead of distribution)  
  > **username_1**: @username_0 Please, provide the link to the examples of the VAE implementations that use the MSE. To give a proper answer, I would need more context.  
  > **username_0**: @username_1, MSE or BCE are the same thing. The VAE paper doesn't talk about comparing the reconstructed image with the original one. Just about optimizing PDFs. See section C.2 in the original paper where they calculate p(x|z) with no MSE / BCE  
  > **username_0**: @username_1, from the article: "for the decoder we used MLPs with either Gaussian or Bernoulli output". That is, the output of the decoder are the parameters of a distribution, not a reconstructed image. We sample from it to get the image.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_0 on 2021-04-18. Score: 6 | Accepted answer
  
  If $p(x|z) \sim \mathcal{N}(f(z), I)$, then
  
  \begin{align}
  \log\ p(x|z)
  &\sim \log\ \exp(-(x-f(z))^2) \\
  &\sim -(x-f(z))^2 \\
  &= -(x-\hat{x})^2,
  \end{align}
  
  where $\hat{x}$, the reconstructed image, is just the distribution mean $f(z)$.
  
  It also makes sense to use the distribution mean when using the decoder (vs. just when training), as it is the one with the highest pdf value. So, the decoder produces a distribution from which we take the mean as our result.
  
  > **Comments:**
  > **username_1**: This seems correct to me (and this is what I had in mind when I wrote my answer above, although I didn't write the derivation), but note that the second $\sim$ is exactly $=$. The first one is correct because you're omitting the denominator of the Gaussian density (which becomes a constant if you optimize wrt $\theta$), so you use approximately equal, which is correct. Moreover, it might be a good idea to add a underscript to $f$, i.e. $f_\theta$ to emphasize that you will be optimizing with respect to those parameters, although you jointly optimize them with $\phi$ (the encoder parameters).  
  
  ---
  
  ### Answer by username_1 on 2021-04-16. Score: 3
  
  On page 5 of [the VAE paper](https://arxiv.org/pdf/1312.6114.pdf#page=5), it's clearly stated
  
  > We let $p\_{\boldsymbol{\theta}}(\mathbf{x} \mid \mathbf{z})$ be a multivariate Gaussian (in case of real-valued data) or Bernoulli (in case of binary data) whose distribution parameters are computed from $\mathbf{z}$ with a MLP (a fully-connected neural network with a single hidden layer, see appendix $\mathrm{C}$ ).
  >
  > ...
  >
  > As explained above and in appendix $\mathrm{C}$, the decoding term $\log p\_{\boldsymbol{\theta}}\left(\mathbf{x}^{(i)} \mid \mathbf{z}^{(i, l)}\right)$ is a Bernoulli or Gaussian MLP, **depending on the type of data we are modelling**.
  
  So, if you are trying to predict real numbers (in the case of images, these can be the RGB values in the range $[0, 1]$), then you can assume $p\_{\boldsymbol{\theta}}(\mathbf{x} \mid \mathbf{z})$ is a Gaussian.
  
  It turns out that maximising the Gaussian likelihood is equivalent to minimising the MSE between the prediction of the decoder and the real image. You can easily show this: just replace $p\_{\boldsymbol{\theta}}(\mathbf{x} \mid \mathbf{z})$ with [the Gaussian pdf](https://en.wikipedia.org/wiki/Normal_distribution), then maximise that wrt the parameters, and you should end up with something that resembles the MSE. G. Hinton shows this in [this video lesson](https://www.youtube.com/watch?v=vEPQNwxd1Y4). See also [this related answer](https://ai.stackexchange.com/a/17671/2444).
  
  So, **yes, minimizing the MSE is theoretically founded**, provided that you're trying to predict some real number.
  
  When the binary cross-entropy (instead of the MSE) is used (e.g. [here](https://github.com/pytorch/examples/blob/main/vae/main.py)), the assumption is that you're maximizing a Bernoulli likelihood (instead of a Gaussian) - this can also be easily shown.
  
  > **Comments:**
  > **username_0**: the pdf is $\frac{1}{\sigma_z \sqrt{2\pi} } e^{-\frac{1}{2}\left(\frac{x-\mu_z}{\sigma_z}\right)^2}$. MSE is $\|\hat{x}-x\|^2$. I don't see the connection other than the fact there's an $(x-\mu_z)^2$ at the power of e. Note that in p(x|z), the original x is used, not the reconstruction.  
  > **username_0**: See section C.2 in the original paper where they calculate p(x|z) with no MSE / BCE  
  > **username_1**: Note sure why you say "Note that in p(x|z), the original x is used, not the reconstruction.". Both the original and the reconstructed images are used. z is used to compute the reconstructed image (or pixel).  
  
  ---
  
  