Answers.Body: <blockquote>
  <p>What I can guess here is that, in VAEs, we assume <span class="math-container">$p(z)$</span> (prior), so we are able to calculate <span class="math-container">$p(x \mid z)$</span>, but for <span class="math-container">$p(x)$</span> we can't assume its distribution? Is it right?</p>
  </blockquote>
  <p>You could assume <span class="math-container">$p(x)$</span> is some distribution (e.g. the Gaussian <span class="math-container">$\mathcal{N}(0, 1)$</span>), but that assumption might be completely wrong - you can also check that against your given data (i.e. if your data is unlikely under your assumed distribution, then maybe your assumption is unreasonable - see <a href="https://statisticsbyjim.com/hypothesis-testing/identify-distribution-data/" rel="nofollow noreferrer">this</a> or <a href="https://en.wikipedia.org/wiki/Normality_test" rel="nofollow noreferrer">this</a>).</p>
  <p>The ultimate goal of a generative model, like the VAE, is to be able to generate data (so you want to use it for tasks like image denoising).
  So, you want to learn a probability distribution that approximates the distribution from which your given dataset was sampled that would allow you to sample more similar data. In practice, you want to learn a probability distribution of the form <span class="math-container">$p(x, z)$</span>, where <span class="math-container">$z$</span> is some latent random variable, which we assume to be involved in the process of generating <span class="math-container">$x$</span> (a random variable that represents the type of data that you have), i.e. <span class="math-container">$x$</span> depends on <span class="math-container">$z$</span>.</p>
  <p>By definition, <span class="math-container">$p(x, z)$</span> can be written as <span class="math-container">$p(x, z) = p(x \mid z)p(z)$</span> or <span class="math-container">$p(x, z) = p(z \mid x) p(x)$</span>. If you put together these equations, you obtain the Bayes rule/theorem <span class="math-container">$p(z \mid x) = \frac{p(x \mid z)p(z)}{p(x)}$</span>, where <span class="math-container">$p(z \mid x)$</span> is the posterior, <span class="math-container">$p(x \mid z)$</span> the likelihood, <span class="math-container">$p(z)$</span> the prior, and <span class="math-container">$p(x)$</span> the marginal (likelihood) or evidence.</p>
  <p>In the VAE, we use the variational distribution, <span class="math-container">$q_\phi$</span>, to approximate <span class="math-container">$p_\theta(z \mid x)$</span>, but we also learn <span class="math-container">$p_\theta(x \mid z)$</span> - in fact, that's the decoder - and we learn it jointly with the variational distribution by maximizing the ELBO. We maximize the ELBO because it's a lower bound on <span class="math-container">$p(x)$</span>, so, by maximizing it, we're also maximizing the marginal likelihood (aka evidence) <span class="math-container">$p(x)$</span>, i.e. we're trying to find the parameters <span class="math-container">$\theta$</span> and <span class="math-container">$\phi$</span>, such that <span class="math-container">$x$</span> is more likely to have been sample from <span class="math-container">$p(x)$</span>, which is unknown. See <a href="https://arxiv.org/pdf/1312.6114.pdf" rel="nofollow noreferrer">the VAE paper</a> (in particular, algorithm 1) for more details.</p>
  <p>Once the VAE is trained, you have <span class="math-container">$q_\tilde{\phi}(z \mid x)$</span> and <span class="math-container">$p_\tilde{\theta}(x \mid z)$</span>. If <span class="math-container">$p_\theta(z \mid x)$</span> is the posterior, then <span class="math-container">$q_\tilde{\phi}(z \mid x)$</span> is an approximation of the posterior. Now, you can generate data with the VAE as follows</p>
  <ol>
  <li>Let <span class="math-container">$q_\tilde{\phi}(z \mid x)$</span> be the new prior (as opposed to the user-defined <span class="math-container">$p(z)$</span>, which was used to train the VAE) - in Bayes statistics, the posterior then becomes the prior (it's our new belief of how <span class="math-container">$z$</span> behaves given our data <span class="math-container">$x$</span>)</li>
  <li>Sample <span class="math-container">$z^i$</span> from <span class="math-container">$q_\tilde{\phi}(z \mid x)$</span></li>
  <li>Sample <span class="math-container">$x^i$</span> from <span class="math-container">$p_\tilde{\theta}(x \mid z^i)$</span></li>
  </ol>
  <p>So, together, <span class="math-container">$\color{blue}{p_\tilde{\theta}(x \mid z^i)}$</span> and <span class="math-container">$\color{red}{q_\tilde{\phi}(z \mid x)}$</span> approximate <span class="math-container">$p(x, z) = \color{blue}{p(x \mid z)} \color{red}{p(z)}$</span>.</p>
  <p>It might be possible to come up with a new objective function (different from the ELBO), where we use some kind of variational distribution to approximate <span class="math-container">$p_\theta(x \mid z)$</span>, but, in the VAE, that's not the case.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-05T10:56:24.970Z
Answers.LastActivityDate: 2022-06-05T12:09:23.103Z
Answers.LastEditDate: 2022-06-05T12:09:23.103Z
Answers.UserDisplayName: nbro
Body: <p>In the VAE, we approximate the probability distribution <span class="math-container">$p(z \mid x)$</span>, where <span class="math-container">$z$</span> is the latent vector and <span class="math-container">$x$</span> is our data. The reason is that <span class="math-container">$p(z \mid x)$</span> becomes impossible to calculate for continuous data because of <span class="math-container">$p(x)$</span>, which require integration (not in closed form) to be solved.</p>
  <p>But why don't we also need to approximate <span class="math-container">$p(x \mid z)$</span>?</p>
  <p>What I can guess here is that, in VAEs, we assume <span class="math-container">$p(z)$</span> (prior), so we are able to calculate <span class="math-container">$p(x \mid z)$</span>, but for <span class="math-container">$p(x)$</span> we can't assume its distribution? Is it right?</p>
  
Tags: <generative-model><variational-autoencoder><probability-theory><variational-inference>
Title: Why don't we also need to approximate $p(x \mid z)$ in the VAE?
UserDisplayName: Nervous Hero
ThreadText: # Why don't we also need to approximate $p(x \mid z)$ in the VAE?
  
  (Asked by: username_0 on 2022-06-02)
  
  In the VAE, we approximate the probability distribution $p(z \mid x)$, where $z$ is the latent vector and $x$ is our data. The reason is that $p(z \mid x)$ becomes impossible to calculate for continuous data because of $p(x)$, which require integration (not in closed form) to be solved.
  
  But why don't we also need to approximate $p(x \mid z)$?
  
  What I can guess here is that, in VAEs, we assume $p(z)$ (prior), so we are able to calculate $p(x \mid z)$, but for $p(x)$ we can't assume its distribution? Is it right?
  
  **Tags:** generative-model, variational-autoencoder, probability-theory, variational-inference
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-06-05. Score: 4 | Accepted answer
  
  > What I can guess here is that, in VAEs, we assume $p(z)$ (prior), so we are able to calculate $p(x \mid z)$, but for $p(x)$ we can't assume its distribution? Is it right?
  
  You could assume $p(x)$ is some distribution (e.g. the Gaussian $\mathcal{N}(0, 1)$), but that assumption might be completely wrong - you can also check that against your given data (i.e. if your data is unlikely under your assumed distribution, then maybe your assumption is unreasonable - see [this](https://statisticsbyjim.com/hypothesis-testing/identify-distribution-data/) or [this](https://en.wikipedia.org/wiki/Normality_test)).
  
  The ultimate goal of a generative model, like the VAE, is to be able to generate data (so you want to use it for tasks like image denoising).
  So, you want to learn a probability distribution that approximates the distribution from which your given dataset was sampled that would allow you to sample more similar data. In practice, you want to learn a probability distribution of the form $p(x, z)$, where $z$ is some latent random variable, which we assume to be involved in the process of generating $x$ (a random variable that represents the type of data that you have), i.e. $x$ depends on $z$.
  
  By definition, $p(x, z)$ can be written as $p(x, z) = p(x \mid z)p(z)$ or $p(x, z) = p(z \mid x) p(x)$. If you put together these equations, you obtain the Bayes rule/theorem $p(z \mid x) = \frac{p(x \mid z)p(z)}{p(x)}$, where $p(z \mid x)$ is the posterior, $p(x \mid z)$ the likelihood, $p(z)$ the prior, and $p(x)$ the marginal (likelihood) or evidence.
  
  In the VAE, we use the variational distribution, $q\_\phi$, to approximate $p\_\theta(z \mid x)$, but we also learn $p\_\theta(x \mid z)$ - in fact, that's the decoder - and we learn it jointly with the variational distribution by maximizing the ELBO. We maximize the ELBO because it's a lower bound on $p(x)$, so, by maximizing it, we're also maximizing the marginal likelihood (aka evidence) $p(x)$, i.e. we're trying to find the parameters $\theta$ and $\phi$, such that $x$ is more likely to have been sample from $p(x)$, which is unknown. See [the VAE paper](https://arxiv.org/pdf/1312.6114.pdf) (in particular, algorithm 1) for more details.
  
  Once the VAE is trained, you have $q\_\tilde{\phi}(z \mid x)$ and $p\_\tilde{\theta}(x \mid z)$. If $p\_\theta(z \mid x)$ is the posterior, then $q\_\tilde{\phi}(z \mid x)$ is an approximation of the posterior. Now, you can generate data with the VAE as follows
  
  1. Let $q\_\tilde{\phi}(z \mid x)$ be the new prior (as opposed to the user-defined $p(z)$, which was used to train the VAE) - in Bayes statistics, the posterior then becomes the prior (it's our new belief of how $z$ behaves given our data $x$)
  2. Sample $z^i$ from $q\_\tilde{\phi}(z \mid x)$
  3. Sample $x^i$ from $p\_\tilde{\theta}(x \mid z^i)$
  
  So, together, $\color{blue}{p\_\tilde{\theta}(x \mid z^i)}$ and $\color{red}{q\_\tilde{\phi}(z \mid x)}$ approximate $p(x, z) = \color{blue}{p(x \mid z)} \color{red}{p(z)}$.
  
  It might be possible to come up with a new objective function (different from the ELBO), where we use some kind of variational distribution to approximate $p\_\theta(x \mid z)$, but, in the VAE, that's not the case.
  
  
  ---
  
  