Answers.Body: <p>Though the posterior <span class="math-container">$p(z|x)$</span> is intractable, we can still compute <span class="math-container">$p(x,z)=p(z)p(x|z)$</span> using conditional probability. As you claimed correctly <span class="math-container">$p(z)$</span> is assumed a simple and known prior distribution such as a standard normal distribution. Depending on the underlying generative process <span class="math-container">$p(x|z)$</span> is often assumed to be a multivariate Gaussian distribution with a diagonal covariance matrix and is typically implemented using a neural network known as decoder which takes the latent variables <span class="math-container">$z$</span> as input and outputs the parameters <span class="math-container">$\theta$</span> of <span class="math-container">$p(x|z)$</span>.</p>
  <p>Your ELBO seems not exactly correct compared to the <a href="https://en.wikipedia.org/wiki/Variational_autoencoder#Evidence_lower_bound_(ELBO)" rel="nofollow noreferrer">reference</a>: <span class="math-container">$${L_{\theta ,\phi }(x):=\mathbb {E} _{z\sim q_{\phi }(\cdot |x)}\left[\ln {\frac {p_{\theta }(x,z)}{q_{\phi }({z|x})}}\right]=\ln p_{\theta }(x)-D_{KL}(q_{\phi }({\cdot |x})\parallel p_{\theta }({\cdot |x}))}$$</span></p>
  <blockquote>
  <p>The form given is not very convenient for maximization, but the following, equivalent form, is: <span class="math-container">$${L_{\theta ,\phi }(x)=\mathbb {E} _{z\sim q_{\phi }(\cdot |x)}\left[\ln p_{\theta }(x|z)\right]-D_{KL}(q_{\phi }({\cdot |x})\parallel p_{\theta }(\cdot ))}$$</span>
  where <span class="math-container">${\ln p_{\theta }(x|z)}$</span> is implemented as <span class="math-container">${-{\frac {1}{2}}\|x-D_{\theta }(z)\|_{2}^{2}}$</span>, since that is, up to an additive constant, what <span class="math-container">${x\sim {\mathcal {N}}(D_{\theta }(z),I)}$</span> yields. That is, we model the distribution of <span class="math-container">$x$</span> conditional on <span class="math-container">$z$</span> to be a Gaussian distribution centered on <span class="math-container">${D_{\theta }(z)}$</span>.</p>
  </blockquote>
  <p>Since distribution of the approximate posterior <span class="math-container">${q_{\phi }(z|x)}$</span> and the prior <span class="math-container">${p_{\theta }(z)}$</span> are often chosen to be Gaussians, ELBO in the reference is finally tractable as <span class="math-container">$${L_{\theta ,\phi }(x)=-{\frac {1}{2}}\mathbb {E} _{z\sim q_{\phi }(\cdot |x)}\left[\|x-D_{\theta }(z)\|_{2}^{2}\right]-{\frac {1}{2}}\left(N\sigma _{\phi }(x)^{2}+\|E_{\phi }(x)\|_{2}^{2}-2N\ln \sigma _{\phi }(x)\right)+Const}$$</span></p>
  <blockquote>
  <p>Here <span class="math-container">$N$</span> is the dimension of <span class="math-container">$z$</span>.</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-21T00:29:59.430Z
Answers.Text: Thank you for you answer. But how can we automatically say that we can model p(x|z) with a decoder, but we can't do the same with p(z|x). Through Bayes formula isn't p(x|z) also dependent on p(x)?
Answers.UserDisplayName: Kiran Manicka
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-21T06:29:39.580Z
Answers.Text: Of course encoder is used to approximate parameters of p(z|x), similar to decoder p(x|z). As implicated in my answer p(z|x)p(x)=p(x|z)p(z) and RHS terms are both tractable, so what I mean p(z|x) is intractable (analytically) is due to intractability of p(x). If you were able to directly learn a parametric model of the evidence p(x) (an integral expression per chain rule as a GMM in my ref) from first principles such as MLE or MAP, then you can simply generate output by sampling p(x). The difficulty of p(x) is that integration in high dimensional latent space is intractable even numerically.
Answers.UserDisplayName: cinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-21T07:44:26.993Z
Answers.Text: From what I am understanding the reason why p(x|z) is tractable is because we simply choose for it to tractable my modeling it with a simply tractable distribution like Gaussian. Then obviously because of p(x), p(z|x) becomes intractable and we have to approximate it with VI. Theoretically, is it possible to choose a simple tractable distribution for p(z|x) instead which would force us to VI approximation on the likelihood?
Answers.UserDisplayName: Kiran Manicka
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-18T18:16:23.630Z
Answers.Text: You have constraints regarding the simple tractable p(z|x) since as posterior it must be close to its prior p(z) which is typically Gaussian. And due to the apparent conjugate prior property of both Gaussian p(z) and likelihood p(x|z), p(z|x) must be Gaussian, no simpler distribution is possible.
Answers.UserDisplayName: cinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-19T05:40:25.113Z
Answers.LastActivityDate: 2024-02-19T05:40:25.113Z
Answers.UserDisplayName: cinch
Body: <p>I'm studying Variational Autoencoders and a lot of the literature says that the posterior is intractable because the marginal distribution p(x) is intractable since the space of z is so large we cannot possibly integrate over it all. So to avoid this they create a lower bound on the log likelihood, the ELBO, which they then try to maximize. The term for the ELBO is:</p>
  <p><span class="math-container">$$E_q[log~ p(z,x)]-E_q[log~q(z)]$$</span></p>
  <p>What I am trying to understand is how is this now tractable. The expectations in the ELBO are still over the distribution of q. Take the first term for example:</p>
  <p><span class="math-container">$$E_q[log~ p(z,x)]=\int_{}^{}q(z)~log~p(z,x)dz$$</span></p>
  <p>Is this not still an integral over all z values? How did we make this problem any more tractable by finding the ELBO?</p>
  <p>Additional Question:
  Also another thing I was confused about is we always say the posterior p(z|x) is not computable because we don't have p(x), but how exactly do we have the numerator, p(x,z).
  <span class="math-container">$$p(z|x)=\frac{p(x,z)}{p(x)}$$</span></p>
  <p>Is this because we assume a prior, and then also assume that we can model p(x|z) with a decoder?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-02-19T12:13:57.293Z
Comments.Text: Please, limit your posts to just one specific question. If you have multiple questions, even if they are related to the same main topic, you can always create multiple posts.
Comments.UserDisplayName: nbro
Tags: <variational-autoencoder><probability-distribution><evidence-lower-bound><bayesian-neural-networks><variational-inference>
Title: How does using the ELBO in VAEs make the problem tractable?
UserDisplayName: Kiran Manicka
ThreadText: # How does using the ELBO in VAEs make the problem tractable?
  
  (Asked by: username_0 on 2024-02-19)
  
  I'm studying Variational Autoencoders and a lot of the literature says that the posterior is intractable because the marginal distribution p(x) is intractable since the space of z is so large we cannot possibly integrate over it all. So to avoid this they create a lower bound on the log likelihood, the ELBO, which they then try to maximize. The term for the ELBO is:
  
  $$E\_q[log~ p(z,x)]-E\_q[log~q(z)]$$
  
  What I am trying to understand is how is this now tractable. The expectations in the ELBO are still over the distribution of q. Take the first term for example:
  
  $$E\_q[log~ p(z,x)]=\int\_{}^{}q(z)~log~p(z,x)dz$$
  
  Is this not still an integral over all z values? How did we make this problem any more tractable by finding the ELBO?
  
  Additional Question:
  Also another thing I was confused about is we always say the posterior p(z|x) is not computable because we don't have p(x), but how exactly do we have the numerator, p(x,z).
  $$p(z|x)=\frac{p(x,z)}{p(x)}$$
  
  Is this because we assume a prior, and then also assume that we can model p(x|z) with a decoder?
  
  **Tags:** variational-autoencoder, probability-distribution, evidence-lower-bound, bayesian-neural-networks, variational-inference
  > **Comments:**
  > **username_1**: Please, limit your posts to just one specific question. If you have multiple questions, even if they are related to the same main topic, you can always create multiple posts.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2024-02-19. Score: 0
  
  Though the posterior $p(z|x)$ is intractable, we can still compute $p(x,z)=p(z)p(x|z)$ using conditional probability. As you claimed correctly $p(z)$ is assumed a simple and known prior distribution such as a standard normal distribution. Depending on the underlying generative process $p(x|z)$ is often assumed to be a multivariate Gaussian distribution with a diagonal covariance matrix and is typically implemented using a neural network known as decoder which takes the latent variables $z$ as input and outputs the parameters $\theta$ of $p(x|z)$.
  
  Your ELBO seems not exactly correct compared to the [reference](https://en.wikipedia.org/wiki/Variational_autoencoder#Evidence_lower_bound_(ELBO)): $${L\_{\theta ,\phi }(x):=\mathbb {E} \_{z\sim q\_{\phi }(\cdot |x)}\left[\ln {\frac {p\_{\theta }(x,z)}{q\_{\phi }({z|x})}}\right]=\ln p\_{\theta }(x)-D\_{KL}(q\_{\phi }({\cdot |x})\parallel p\_{\theta }({\cdot |x}))}$$
  
  > The form given is not very convenient for maximization, but the following, equivalent form, is: $${L\_{\theta ,\phi }(x)=\mathbb {E} \_{z\sim q\_{\phi }(\cdot |x)}\left[\ln p\_{\theta }(x|z)\right]-D\_{KL}(q\_{\phi }({\cdot |x})\parallel p\_{\theta }(\cdot ))}$$
  > where ${\ln p\_{\theta }(x|z)}$ is implemented as ${-{\frac {1}{2}}\|x-D\_{\theta }(z)\|\_{2}^{2}}$, since that is, up to an additive constant, what ${x\sim {\mathcal {N}}(D\_{\theta }(z),I)}$ yields. That is, we model the distribution of $x$ conditional on $z$ to be a Gaussian distribution centered on ${D\_{\theta }(z)}$.
  
  Since distribution of the approximate posterior ${q\_{\phi }(z|x)}$ and the prior ${p\_{\theta }(z)}$ are often chosen to be Gaussians, ELBO in the reference is finally tractable as $${L\_{\theta ,\phi }(x)=-{\frac {1}{2}}\mathbb {E} \_{z\sim q\_{\phi }(\cdot |x)}\left[\|x-D\_{\theta }(z)\|\_{2}^{2}\right]-{\frac {1}{2}}\left(N\sigma \_{\phi }(x)^{2}+\|E\_{\phi }(x)\|\_{2}^{2}-2N\ln \sigma \_{\phi }(x)\right)+Const}$$
  
  > Here $N$ is the dimension of $z$.
  
  > **Comments:**
  > **username_0**: Thank you for you answer. But how can we automatically say that we can model p(x|z) with a decoder, but we can't do the same with p(z|x). Through Bayes formula isn't p(x|z) also dependent on p(x)?  
  > **username_2**: Of course encoder is used to approximate parameters of p(z|x), similar to decoder p(x|z). As implicated in my answer p(z|x)p(x)=p(x|z)p(z) and RHS terms are both tractable, so what I mean p(z|x) is intractable (analytically) is due to intractability of p(x). If you were able to directly learn a parametric model of the evidence p(x) (an integral expression per chain rule as a GMM in my ref) from first principles such as MLE or MAP, then you can simply generate output by sampling p(x). The difficulty of p(x) is that integration in high dimensional latent space is intractable even numerically.  
  > **username_0**: From what I am understanding the reason why p(x|z) is tractable is because we simply choose for it to tractable my modeling it with a simply tractable distribution like Gaussian. Then obviously because of p(x), p(z|x) becomes intractable and we have to approximate it with VI. Theoretically, is it possible to choose a simple tractable distribution for p(z|x) instead which would force us to VI approximation on the likelihood?  
  > **username_2**: You have constraints regarding the simple tractable p(z|x) since as posterior it must be close to its prior p(z) which is typically Gaussian. And due to the apparent conjugate prior property of both Gaussian p(z) and likelihood p(x|z), p(z|x) must be Gaussian, no simpler distribution is possible.  
  
  ---
  
  