Answers.Body: <blockquote>
  <p>My question is: what exactly is <span class="math-container">$\log p(x|z,w)$</span>? I don't know what this distribution is exactly, so does one make any assumption about the distribution of x?</p>
  </blockquote>
  <p>I completely understand the question. I think the main problem of Variational Autoencoders is that the selected notation for explaining the math is between Probability theory notation and standard Machine Learning notation. To understand the concepts from it, it can be somewhat confusing. What exactly is <span class="math-container">$\log p(x|z,w)$</span>? Well that part is called the reconstruction error of the ELBO estimation. Let's see it in parts to fully understand it.</p>
  <p>Inside the natural logarithm there is the probability <span class="math-container">$p(x|z,w)$</span>. This is called the posterior of the decoding phase. This is the probability of getting <span class="math-container">$x$</span> given <span class="math-container">$z$</span> with the parameters <span class="math-container">$w$</span> that are the parameters of the decoder network. Then the problem is, what exactly are <span class="math-container">$z$</span> and <span class="math-container">$x$</span> given that we are working with images?</p>
  <p>Well, <span class="math-container">$z$</span> are the parameters for the Gaussian distribution generated by the encoder. <span class="math-container">$z$</span> are two vectors - the standard deviation vector and the mean vector. These parameters generated by the encoder are then used to calculate the posterior of the decoder <span class="math-container">$p(x|z)$</span>, which is the probability of generating the original image given <span class="math-container">$z$</span>. and <span class="math-container">$x$</span> is the data we give as inputs.(1 footnote).</p>
  <p>After doing the reparameterization trick:</p>
  <p><span class="math-container">$z = \mu + \sigma \odot \epsilon$</span> where <span class="math-container">$\epsilon \sim \mathcal{N}(0,1)$</span></p>
  <p>That is just a smart way of sampling from a distribution, in this case a Normal distribution (2 footnote).</p>
  <p>Then that <span class="math-container">$z$</span> vector is passed through the decoder for adding the highly nonlinear distribution at the end and output a probability. Now the question is, what is the form of that probability distribution? In this case we used a sigmoidal function for outputting a Bernoulli or binomial distribution (3 footnote). Why do we choose the sigmoid function? Because first, all the pixels in the generated image must be independent, meaning that when assigning a probability to a pixel we don't want others to influence them like in the case of a softmax function. A second reason is that the sigmoid function has a range from 0 to 1 and when working with images the pixels are bounded from 0 to 255, meaning that we can normalize all the data at the beginning and the inductive bias is more powerful.</p>
  <blockquote>
  <p>In the original paper (Auto-Encoding Variational Bayes), they claim to use MLP gaussian output for the decoder. What exactly does that mean?</p>
  </blockquote>
  <p>Citing the paper it says:</p>
  <blockquote>
  <p>&quot;The decoding distribution <span class="math-container">$p_\theta(x^{(i)}|z^{(i)})$</span> is a Bernoulli or Gaussian MLP, depending on the type of data we are modelling.&quot;</p>
  </blockquote>
  <p>This means that the decoder network has all activation functions consist of sigmoidal or GELU functions. This was terminology that was very common in the early times of Machine Learning. Now most likely you would not hear the term Bernoulli MLP or Gaussian MLP because most neural networks today consist of various activation functions.</p>
  <p>Footnotes:</p>
  <ol>
  <li><p>Although they used mean sum in this paper, they really just using the ELBO estimation for calculating the loss. The thing is that ELBO uses the log of the expectation of a random variable. Binary Cross Entropy also uses the expectation of a random variable. The reason we can use BCE in the loss is because we interpret the original data as a probability and because the final output is also a probability we can perform BCE knowing the probability of each data point. As the contrary of using a mean sum that is making the assumption that the probabilities of all the data are the same because we don't have real information of the distribution of the original data.</p>
  </li>
  <li><p>The reason we pick a Normal Distribution is because it is a very known distribution which we can operate with. For example it is very easy for us to calculate the KL Divergence on the loss function using a Normal Distribution.</p>
  </li>
  <li><p>Technically if we try to be as theoretical as possible we should pick the Gaussian distribution with the GELU activation function but because of the advantages of using sigmoid for image data we use a Bernoulli distribution with sigmoid. Another thing is that most of the time the fully theoretical formula doesn't work as well as nitpicking with different architectural decisions. What I am trying to say is that I have seen VAEs being implemented with MSE loss functions even when it is not a regression problem but a classification one. So when implementing an efficient algorithm for a given data you can be less strict with the theory for gaining more performance at the expense of having a lack of theoretical background.</p>
  </li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-12T18:57:23.837Z
Answers.Text: In my understanding, as $p(x|z,w)$ being the decoder then it is not a probability distribution, but a function of $z$ parametrized by $w$, because the decoder network is deterministic, why picking GeLu or sigmoid makes it a well defined probability distribution, and more than that, a Gaussian distribution? Is that a simple convenient assumption to introduce the MSE loss?
Answers.UserDisplayName: piero
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-12T19:26:08.453Z
Answers.Text: If you Assume $p(x|z,w)$ is normal with mean $decoder(z)$ and covariance equals to identity, then its log takes the form of MSE loss, but this does not make any sense, its a nonsense assumption.
Answers.UserDisplayName: piero
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-10T23:56:41.263Z
Answers.LastActivityDate: 2024-01-10T23:56:41.263Z
Answers.UserDisplayName: Cesar Ruiz
Body: <p>From Bishop's recent book on Deep Learning, it says the ELBO for Gaussian latent prior can be approximated by</p>
  <p><span class="math-container">$\frac{1}{L}\sum_{l=1}^L \ln p(x_n|z_n^l,w) + KL(q(z_n|x_n,\phi)||p(z_n))$</span></p>
  <p>where <span class="math-container">$n$</span> are data samples, and the KL term have a closed form for gaussian prior p(z). My question is: what exactly is <span class="math-container">$\ln p(x_n | z_n^l,w)$</span>? I don't know what this distribution is exactly, so one makes any assumption about the distribution of <span class="math-container">$x$</span>?</p>
  <p>update:</p>
  <p>In the original paper (Auto-Encoding Variational Bayes), they claim to use MLP gaussian output for the decoder. What exactly that means?</p>
  
Tags: <variational-autoencoder><latent-variable>
Title: Variational Lower Bound in VAE for Gaussian latent prior
UserDisplayName: piero
ThreadText: # Variational Lower Bound in VAE for Gaussian latent prior
  
  (Asked by: username_0 on 2024-01-08)
  
  From Bishop's recent book on Deep Learning, it says the ELBO for Gaussian latent prior can be approximated by
  
  $\frac{1}{L}\sum\_{l=1}^L \ln p(x\_n|z\_n^l,w) + KL(q(z\_n|x\_n,\phi)||p(z\_n))$
  
  where $n$ are data samples, and the KL term have a closed form for gaussian prior p(z). My question is: what exactly is $\ln p(x\_n | z\_n^l,w)$? I don't know what this distribution is exactly, so one makes any assumption about the distribution of $x$?
  
  update:
  
  In the original paper (Auto-Encoding Variational Bayes), they claim to use MLP gaussian output for the decoder. What exactly that means?
  
  **Tags:** variational-autoencoder, latent-variable
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-01-10. Score: 1
  
  > My question is: what exactly is $\log p(x|z,w)$? I don't know what this distribution is exactly, so does one make any assumption about the distribution of x?
  
  I completely understand the question. I think the main problem of Variational Autoencoders is that the selected notation for explaining the math is between Probability theory notation and standard Machine Learning notation. To understand the concepts from it, it can be somewhat confusing. What exactly is $\log p(x|z,w)$? Well that part is called the reconstruction error of the ELBO estimation. Let's see it in parts to fully understand it.
  
  Inside the natural logarithm there is the probability $p(x|z,w)$. This is called the posterior of the decoding phase. This is the probability of getting $x$ given $z$ with the parameters $w$ that are the parameters of the decoder network. Then the problem is, what exactly are $z$ and $x$ given that we are working with images?
  
  Well, $z$ are the parameters for the Gaussian distribution generated by the encoder. $z$ are two vectors - the standard deviation vector and the mean vector. These parameters generated by the encoder are then used to calculate the posterior of the decoder $p(x|z)$, which is the probability of generating the original image given $z$. and $x$ is the data we give as inputs.(1 footnote).
  
  After doing the reparameterization trick:
  
  $z = \mu + \sigma \odot \epsilon$ where $\epsilon \sim \mathcal{N}(0,1)$
  
  That is just a smart way of sampling from a distribution, in this case a Normal distribution (2 footnote).
  
  Then that $z$ vector is passed through the decoder for adding the highly nonlinear distribution at the end and output a probability. Now the question is, what is the form of that probability distribution? In this case we used a sigmoidal function for outputting a Bernoulli or binomial distribution (3 footnote). Why do we choose the sigmoid function? Because first, all the pixels in the generated image must be independent, meaning that when assigning a probability to a pixel we don't want others to influence them like in the case of a softmax function. A second reason is that the sigmoid function has a range from 0 to 1 and when working with images the pixels are bounded from 0 to 255, meaning that we can normalize all the data at the beginning and the inductive bias is more powerful.
  
  > In the original paper (Auto-Encoding Variational Bayes), they claim to use MLP gaussian output for the decoder. What exactly does that mean?
  
  Citing the paper it says:
  
  > "The decoding distribution $p\_\theta(x^{(i)}|z^{(i)})$ is a Bernoulli or Gaussian MLP, depending on the type of data we are modelling."
  
  This means that the decoder network has all activation functions consist of sigmoidal or GELU functions. This was terminology that was very common in the early times of Machine Learning. Now most likely you would not hear the term Bernoulli MLP or Gaussian MLP because most neural networks today consist of various activation functions.
  
  Footnotes:
  
  1. Although they used mean sum in this paper, they really just using the ELBO estimation for calculating the loss. The thing is that ELBO uses the log of the expectation of a random variable. Binary Cross Entropy also uses the expectation of a random variable. The reason we can use BCE in the loss is because we interpret the original data as a probability and because the final output is also a probability we can perform BCE knowing the probability of each data point. As the contrary of using a mean sum that is making the assumption that the probabilities of all the data are the same because we don't have real information of the distribution of the original data.
  2. The reason we pick a Normal Distribution is because it is a very known distribution which we can operate with. For example it is very easy for us to calculate the KL Divergence on the loss function using a Normal Distribution.
  3. Technically if we try to be as theoretical as possible we should pick the Gaussian distribution with the GELU activation function but because of the advantages of using sigmoid for image data we use a Bernoulli distribution with sigmoid. Another thing is that most of the time the fully theoretical formula doesn't work as well as nitpicking with different architectural decisions. What I am trying to say is that I have seen VAEs being implemented with MSE loss functions even when it is not a regression problem but a classification one. So when implementing an efficient algorithm for a given data you can be less strict with the theory for gaining more performance at the expense of having a lack of theoretical background.
  
  > **Comments:**
  > **username_0**: In my understanding, as $p(x|z,w)$ being the decoder then it is not a probability distribution, but a function of $z$ parametrized by $w$, because the decoder network is deterministic, why picking GeLu or sigmoid makes it a well defined probability distribution, and more than that, a Gaussian distribution? Is that a simple convenient assumption to introduce the MSE loss?  
  > **username_0**: If you Assume $p(x|z,w)$ is normal with mean $decoder(z)$ and covariance equals to identity, then its log takes the form of MSE loss, but this does not make any sense, its a nonsense assumption.  
  
  ---
  
  