Answers.Body: <p>Let me try to give you a less mathematical, more intuitive explanation.</p>
  <p>You assume a latent space <span class="math-container">$\mathcal{Z}$</span> and you approximate the true distribution <span class="math-container">$p(x)$</span> as:
  <span class="math-container">$$ p_\theta(x) = \sum_\mathcal{Z} p_\theta(x|z) p_Z(z) = \mathbb{E}_{z \sim p_Z} p_\theta(x|z) $$</span>
  Here you use a neural net with parameters <span class="math-container">$\theta$</span> to model <span class="math-container">$p(x|z)$</span>.</p>
  <p>However, there is a problem that arises in this setting. In order to optimize our model we actually want to sample <span class="math-container">$z$</span> that would be a good match for the given data point <span class="math-container">$x$</span>, but there is actually very little chance that that would be the case. If there really exists an underlying latent space to your data, then the best distribution to sample <span class="math-container">$z$</span> from is <span class="math-container">$p(z|x)$</span>. So what you do now is you try to approximate this with a second neural network <span class="math-container">$q_\phi(x)$</span>.</p>
  <p>You want to maximize the log-probability of the training data by optimizing <span class="math-container">$\theta$</span>:</p>
  <p><span class="math-container">\begin{aligned}
  \max_\theta \log p_\theta(x)
  &amp;= \log \mathbb{E}_{z \sim q_\phi(x)} \bigg[ p_\theta(x|z) \frac{p_Z(z)}{q_\phi(z|x)} \bigg] \\
  &amp; \geq \mathbb{E}_{z \sim q_\phi(x)} \log \bigg[ p_\theta(x|z) \frac{p_Z(z)}{q_\phi(z|x)} \bigg] \\
  &amp; = \mathbb{E}_{z \sim q_\phi(x)} \bigg[ \log p_\theta(x) - KL(q_\phi(x) \; || \;p_Z) \bigg]
  \end{aligned}</span></p>
  <p>And you also want to minimize the KL divergence between <span class="math-container">$q_\phi(x)$</span> and <span class="math-container">$p(z|x)$</span> by optimizing <span class="math-container">$\phi$</span>:</p>
  <p><span class="math-container">\begin{aligned}
  \min_\phi KL(q_\phi(z|x) \; || \; p_\theta(z|x) )
  &amp;= \mathbb{E}_{z \sim q_\phi(x)} \log \frac{q_\phi(z|x)}{p_\theta(z|x)} \\
  &amp;= \mathbb{E}_{z \sim q_\phi(x|z)} \Big[ q_\phi(z|x) - p_Z(z) - \log p_\phi(x|z) \Big] + \log p_\theta(x)
  \end{aligned}</span></p>
  <p>Note that the last term in the objective is constant with respect to <span class="math-container">$\phi$</span>, so in the end we are left with the same equation but this time we optimize over <span class="math-container">$\phi$</span>.</p>
  <p>Finally, the objective is:</p>
  <p><span class="math-container">$$
  \max_{\theta, \phi} \mathbb{E}_{z \sim q_\phi(x)} \bigg[ \log p_\theta(x) - KL(q_\phi(x) \; || \;p_Z) \bigg]
  $$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-04T01:01:21.500Z
Answers.LastActivityDate: 2024-03-04T01:01:21.500Z
Answers.UserDisplayName: pi-tau
Answers.Body: <p>Variational methods are designed for situations allowing us to avoid the intractable integral such as <span class="math-container">$p(x)$</span> parameterized by the amortized parameter <span class="math-container">$\theta$</span> along with <span class="math-container">$p(z|x)$</span> by transforming the Bayesian inference problem into an optimization problem wrt <span class="math-container">$\theta$</span> and additional <span class="math-container">$\phi$</span> parameterizing the needed variational <span class="math-container">$q(z|x)$</span>. And since in your reference the marginal evidence <span class="math-container">$p(x)$</span> defined in the latent variable model <em>evaluated</em> at the <span class="math-container">$i$</span>-th example of the training set (note your reference uses <span class="math-container">$p(x^{(i)})$</span> before notation simplification) integrates out <span class="math-container">$p_{\theta}(x|z)$</span> by all possible values of the latent variable <span class="math-container">$z$</span> to induce the said example, it's a (possibly very small) constant wrt the optimized parameters <span class="math-container">$\theta$</span> and <span class="math-container">$\phi$</span> and thus irrelevant of <span class="math-container">$q_{\phi}(z|x)$</span>. But note for your confusion the marginal likelihood <span class="math-container">$p(x)$</span> itself as evidence is essentially a GMM as explained in VAE's <a href="https://en.wikipedia.org/wiki/Variational_autoencoder#Evidence_lower_bound_(ELBO)" rel="nofollow noreferrer">wikipedia reference</a>.</p>
  <p>To further avoid the intractable <span class="math-container">$p(x)$</span> in the ELBO, your reference arrives at the same convenient form of ELBO as mentioned in wikipedia:</p>
  <blockquote>
  <p><span class="math-container">$${L_{\theta ,\phi }(x):=\mathbb {E} _{z\sim q_{\phi }(\cdot |x)}\left[\ln {\frac {p_{\theta }(x,z)}{q_{\phi }({z|x})}}\right]=\ln p_{\theta }(x)-D_{KL}(q_{\phi }({\cdot |x})\parallel p_{\theta }({\cdot |x}))}$$</span>
  The form given is not very convenient for maximization, but the following, equivalent form, is: <span class="math-container">$${L_{\theta ,\phi }(x)=\mathbb {E} _{z\sim q_{\phi }(\cdot |x)}\left[\ln p_{\theta }(x|z)\right]-D_{KL}(q_{\phi }({\cdot |x})\parallel p_{\theta }(\cdot ))}$$</span> where <span class="math-container">${\ln p_{\theta }(x|z)}$</span> is implemented as <span class="math-container">${-{\frac {1}{2}}\|x-D_{\theta }(z)\|_{2}^{2}}$</span>, since that is, up to an additive constant, what <span class="math-container">${x\sim {\mathcal {N}}(D_{\theta }(z),I)}$</span> yields. That is, we model the distribution of <span class="math-container">$x$</span> conditional on <span class="math-container">$z$</span> to be a Gaussian distribution centered on <span class="math-container">${D_{\theta }(z)}$</span>.</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-23T06:26:41.157Z
Answers.Text: Thank you for your answer. If our goal in generative modeling is to maximize p(x), then why does VAE math all center around trying to approximate p(z|x), Is this because with q(z|x), p(z), and p(x|z) all of which we have after variation inference, we can calculate/approximate p(x). Also as a side note, I am confused by the use of theta. When we say p_theta(x), does theta refer to our neural network parameters or parameters to a theoretical unknown distribution that we don't have the analytical form for.
Answers.UserDisplayName: Kiran Manicka
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-23T07:31:32.817Z
Answers.Text: As clearly stated in your own ref, generative modeling goal is mostly about the first 2 problems, while the 3rd problem of prior evidence p(x) modeled as a GMM distribution or evaluated at an example as a constant is much less important (irrelevant to ELBO as explained in my answer). Theta is the non-variational decoder network parameters for the posterior multivariate Gaussian distribution p(x|z) which certainly has analytic form but unknown parameters such as mean vector and covariant matrix, the theta parameterized decoder’s outputs are nothing but the said Gassian’s unknown parameters.
Answers.UserDisplayName: cinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-24T00:57:51.977Z
Answers.Text: so from what I am understanding the in the equation log p(x)=ELBO -KL(q(z|x)||p(z|x)), log p(x) represents the true data generating distribution and does not change because it is not dependent on theta or any parameters that exist within our model. Does this intuition sound right?
Answers.UserDisplayName: Kiran Manicka
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-24T06:57:32.157Z
Answers.Text: Check the line above "*The term logp(x) is a constant*" in your ref, '-' should be "+". Since the logp(x) is constant, then minimizing the KL-Divergence is equivalent o maximizing the ELBO which is the lower bound of log-evidence p(x) as KL-D is always greater than 0. Since x is continuous for image like data, as explained in my answer, here p(x) mean its pdf evaluated at the ith training data which is an intractable integral but we know it must be a constant w.r.t. the learning parameters theta and phi. So your intuition is correct except it's not true data generating distribution meant here.
Answers.UserDisplayName: cinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-09T20:03:40.510Z
Answers.Text: @KiranManicka I recently went through this myself. Is the above clear? The short answer is that if p(z) and p(x|z) are known, then p(x) is well-defined. p(z) is chosen by us, and p(x|z) is fixed for each step of training, and so there is a well-defined p(x) at that step.
Answers.UserDisplayName: A_P
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-22T23:43:49.890Z
Answers.LastActivityDate: 2024-08-01T02:36:11.373Z
Answers.LastEditDate: 2024-08-01T02:36:11.373Z
Answers.UserDisplayName: cinch
Body: <p>I was reading <a href="https://medium.com/@j.zh/mathematics-behind-variational-autoencoders-c69297301957" rel="nofollow noreferrer">the article on medium</a>, where the author cites this equation for Variational Inference:
  <span class="math-container">\begin{align*}
  \text{KL}(q(z|x^{(i)})||p(z|x^{(i)})) &amp;= \int_z q(z|x^{(i)})\text{log}\frac{q(z|x^{(i)})}{p(z|x^{(i)})} dz \\
  &amp;= \mathbb{E}_{||}[\text{log}(q(z|x^{(i)}))] -  \mathbb{E}_{||}[\text{log}(p(z|x^{(i)}))]\\
  &amp;= \mathbb{E}_{||}[\text{log}(q(z|x^{(i)}))] - \mathbb{E}_{||}[\text{log}(p(x^{(i)}, z))] + \mathbb{E}_q[\text{log}(p(x^{(i)}))]\\
  &amp;= \mathbb{E}_{||}[\text{log}(q(z|x^{(i)}))] - \mathbb{E}_{||}[\text{log}(p(x^{(i)}, z))] + \text{log}(p(x^{(i)}))\\
  &amp;= -\text{ELBO} + \text{log}(p(x^{(i)}))\\
  \end{align*}</span></p>
  <p>I understand all of the math behind this equation, but I was wondering what is the underlying intuition behind each of the terms in this equation (KL divergence, ELBO, and logp(x))?</p>
  <p>The author claims that <span class="math-container">$\text{log} p(x)$</span> is a constant in this equation and I'm having a hard time understanding why. Is <span class="math-container">$p(x)$</span> considered to be the theoretical data generating distribution which created our <span class="math-container">$x$</span>'s and not the model that we are training?</p>
  
Tags: <variational-autoencoder><probability-distribution><evidence-lower-bound><latent-variable><variational-inference>
Title: What is the meaning of log p(x) in VAE math and why is it constant
UserDisplayName: Kiran Manicka
ThreadText: # What is the meaning of log p(x) in VAE math and why is it constant
  
  (Asked by: username_0 on 2024-02-22)
  
  I was reading [the article on medium](https://medium.com/@j.zh/mathematics-behind-variational-autoencoders-c69297301957), where the author cites this equation for Variational Inference:
  \begin{align\*}
  \text{KL}(q(z|x^{(i)})||p(z|x^{(i)})) &= \int\_z q(z|x^{(i)})\text{log}\frac{q(z|x^{(i)})}{p(z|x^{(i)})} dz \\
  &= \mathbb{E}\_{||}[\text{log}(q(z|x^{(i)}))] - \mathbb{E}\_{||}[\text{log}(p(z|x^{(i)}))]\\
  &= \mathbb{E}\_{||}[\text{log}(q(z|x^{(i)}))] - \mathbb{E}\_{||}[\text{log}(p(x^{(i)}, z))] + \mathbb{E}\_q[\text{log}(p(x^{(i)}))]\\
  &= \mathbb{E}\_{||}[\text{log}(q(z|x^{(i)}))] - \mathbb{E}\_{||}[\text{log}(p(x^{(i)}, z))] + \text{log}(p(x^{(i)}))\\
  &= -\text{ELBO} + \text{log}(p(x^{(i)}))\\
  \end{align\*}
  
  I understand all of the math behind this equation, but I was wondering what is the underlying intuition behind each of the terms in this equation (KL divergence, ELBO, and logp(x))?
  
  The author claims that $\text{log} p(x)$ is a constant in this equation and I'm having a hard time understanding why. Is $p(x)$ considered to be the theoretical data generating distribution which created our $x$'s and not the model that we are training?
  
  **Tags:** variational-autoencoder, probability-distribution, evidence-lower-bound, latent-variable, variational-inference
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2024-03-04. Score: 0
  
  Let me try to give you a less mathematical, more intuitive explanation.
  
  You assume a latent space $\mathcal{Z}$ and you approximate the true distribution $p(x)$ as:
  $$ p\_\theta(x) = \sum\_\mathcal{Z} p\_\theta(x|z) p\_Z(z) = \mathbb{E}\_{z \sim p\_Z} p\_\theta(x|z) $$
  Here you use a neural net with parameters $\theta$ to model $p(x|z)$.
  
  However, there is a problem that arises in this setting. In order to optimize our model we actually want to sample $z$ that would be a good match for the given data point $x$, but there is actually very little chance that that would be the case. If there really exists an underlying latent space to your data, then the best distribution to sample $z$ from is $p(z|x)$. So what you do now is you try to approximate this with a second neural network $q\_\phi(x)$.
  
  You want to maximize the log-probability of the training data by optimizing $\theta$:
  
  \begin{aligned}
  \max\_\theta \log p\_\theta(x)
  &= \log \mathbb{E}\_{z \sim q\_\phi(x)} \bigg[ p\_\theta(x|z) \frac{p\_Z(z)}{q\_\phi(z|x)} \bigg] \\
  & \geq \mathbb{E}\_{z \sim q\_\phi(x)} \log \bigg[ p\_\theta(x|z) \frac{p\_Z(z)}{q\_\phi(z|x)} \bigg] \\
  & = \mathbb{E}\_{z \sim q\_\phi(x)} \bigg[ \log p\_\theta(x) - KL(q\_\phi(x) \; || \;p\_Z) \bigg]
  \end{aligned}
  
  And you also want to minimize the KL divergence between $q\_\phi(x)$ and $p(z|x)$ by optimizing $\phi$:
  
  \begin{aligned}
  \min\_\phi KL(q\_\phi(z|x) \; || \; p\_\theta(z|x) )
  &= \mathbb{E}\_{z \sim q\_\phi(x)} \log \frac{q\_\phi(z|x)}{p\_\theta(z|x)} \\
  &= \mathbb{E}\_{z \sim q\_\phi(x|z)} \Big[ q\_\phi(z|x) - p\_Z(z) - \log p\_\phi(x|z) \Big] + \log p\_\theta(x)
  \end{aligned}
  
  Note that the last term in the objective is constant with respect to $\phi$, so in the end we are left with the same equation but this time we optimize over $\phi$.
  
  Finally, the objective is:
  
  $$
  \max\_{\theta, \phi} \mathbb{E}\_{z \sim q\_\phi(x)} \bigg[ \log p\_\theta(x) - KL(q\_\phi(x) \; || \;p\_Z) \bigg]
  $$
  
  
  ---
  
  ### Answer by username_2 on 2024-02-22. Score: 0
  
  Variational methods are designed for situations allowing us to avoid the intractable integral such as $p(x)$ parameterized by the amortized parameter $\theta$ along with $p(z|x)$ by transforming the Bayesian inference problem into an optimization problem wrt $\theta$ and additional $\phi$ parameterizing the needed variational $q(z|x)$. And since in your reference the marginal evidence $p(x)$ defined in the latent variable model *evaluated* at the $i$-th example of the training set (note your reference uses $p(x^{(i)})$ before notation simplification) integrates out $p\_{\theta}(x|z)$ by all possible values of the latent variable $z$ to induce the said example, it's a (possibly very small) constant wrt the optimized parameters $\theta$ and $\phi$ and thus irrelevant of $q\_{\phi}(z|x)$. But note for your confusion the marginal likelihood $p(x)$ itself as evidence is essentially a GMM as explained in VAE's [wikipedia reference](https://en.wikipedia.org/wiki/Variational_autoencoder#Evidence_lower_bound_(ELBO)).
  
  To further avoid the intractable $p(x)$ in the ELBO, your reference arrives at the same convenient form of ELBO as mentioned in wikipedia:
  
  > $${L\_{\theta ,\phi }(x):=\mathbb {E} \_{z\sim q\_{\phi }(\cdot |x)}\left[\ln {\frac {p\_{\theta }(x,z)}{q\_{\phi }({z|x})}}\right]=\ln p\_{\theta }(x)-D\_{KL}(q\_{\phi }({\cdot |x})\parallel p\_{\theta }({\cdot |x}))}$$
  > The form given is not very convenient for maximization, but the following, equivalent form, is: $${L\_{\theta ,\phi }(x)=\mathbb {E} \_{z\sim q\_{\phi }(\cdot |x)}\left[\ln p\_{\theta }(x|z)\right]-D\_{KL}(q\_{\phi }({\cdot |x})\parallel p\_{\theta }(\cdot ))}$$ where ${\ln p\_{\theta }(x|z)}$ is implemented as ${-{\frac {1}{2}}\|x-D\_{\theta }(z)\|\_{2}^{2}}$, since that is, up to an additive constant, what ${x\sim {\mathcal {N}}(D\_{\theta }(z),I)}$ yields. That is, we model the distribution of $x$ conditional on $z$ to be a Gaussian distribution centered on ${D\_{\theta }(z)}$.
  
  > **Comments:**
  > **username_0**: Thank you for your answer. If our goal in generative modeling is to maximize p(x), then why does VAE math all center around trying to approximate p(z|x), Is this because with q(z|x), p(z), and p(x|z) all of which we have after variation inference, we can calculate/approximate p(x). Also as a side note, I am confused by the use of theta. When we say p_theta(x), does theta refer to our neural network parameters or parameters to a theoretical unknown distribution that we don't have the analytical form for.  
  > **username_2**: As clearly stated in your own ref, generative modeling goal is mostly about the first 2 problems, while the 3rd problem of prior evidence p(x) modeled as a GMM distribution or evaluated at an example as a constant is much less important (irrelevant to ELBO as explained in my answer). Theta is the non-variational decoder network parameters for the posterior multivariate Gaussian distribution p(x|z) which certainly has analytic form but unknown parameters such as mean vector and covariant matrix, the theta parameterized decoder’s outputs are nothing but the said Gassian’s unknown parameters.  
  > **username_0**: so from what I am understanding the in the equation log p(x)=ELBO -KL(q(z|x)||p(z|x)), log p(x) represents the true data generating distribution and does not change because it is not dependent on theta or any parameters that exist within our model. Does this intuition sound right?  
  > **username_2**: Check the line above "*The term logp(x) is a constant*" in your ref, '-' should be "+". Since the logp(x) is constant, then minimizing the KL-Divergence is equivalent o maximizing the ELBO which is the lower bound of log-evidence p(x) as KL-D is always greater than 0. Since x is continuous for image like data, as explained in my answer, here p(x) mean its pdf evaluated at the ith training data which is an intractable integral but we know it must be a constant w.r.t. the learning parameters theta and phi. So your intuition is correct except it's not true data generating distribution meant here.  
  > **username_3**: @KiranManicka I recently went through this myself. Is the above clear? The short answer is that if p(z) and p(x|z) are known, then p(x) is well-defined. p(z) is chosen by us, and p(x|z) is fixed for each step of training, and so there is a well-defined p(x) at that step.  
  
  ---
  
  