Answers.Body: <p><strong>TLDR:</strong> We're doing a maximum likelihood fit of our model. The VAE sets this up in a way that doesn't require evaluating the model likelihood, but instead expresses a lower bound in terms of reconstruction error.</p>
  <p>Here's an explanation in two parts.</p>
  <h4>1. Variational inference in general</h4>
  <p>We have a generative model  <span class="math-container">$z \rightarrow x$</span> with latent variable <span class="math-container">$z \sim p(z)$</span>, and observations, <span class="math-container">$x$</span>, generated by <span class="math-container">$p(x|z)$</span>. We want to get the posterior <span class="math-container">$p(z|x)$</span>, but this inference may be computationally infeasible even if <span class="math-container">$p(z)$</span> and <span class="math-container">$p(x|z)$</span> are known <em>because <span class="math-container">$p(x)$</span> is hard to evaluate</em>. So, we instead approximate <span class="math-container">$p(z|x)$</span> by <span class="math-container">$q(z)$</span>.</p>
  <p>Denoting the ELBO by <span class="math-container">$\mathcal{L}(x)$</span>, we have (by expanding <span class="math-container">$D_{KL}$</span>):
  <span class="math-container">$$ \log p(x) = D_{KL}\big(q(z) \big|\big| p(z|x) \big) + \mathcal{L}(x). $$</span>
  Note that <span class="math-container">$p(x)$</span> is part of the generative model so, given data <span class="math-container">$x$</span>, <span class="math-container">$p(x)$</span> is <em>constant</em> regardless of the approximation <span class="math-container">$q(z)$</span>. This means that increasing <span class="math-container">$\mathcal{L}(x)$</span> by changing <span class="math-container">$q(z)$</span> is equivalent to decreasing <span class="math-container">$D_{KL}\big(q(z) \big|\big| p(z|x) \big)$</span> and vice versa.</p>
  <p>Thus, the max ELBO approximation is the best in the sense that <span class="math-container">$q(z)$</span> is closest to <span class="math-container">$p(z|x)$</span> in KL terms. This is the classic reasoning for maximizing the ELBO.</p>
  <h4>2. Variational autoencoder</h4>
  <p>For the variational autoencoder model, we</p>
  <ol>
  <li>choose <span class="math-container">$p(z)$</span> for the latent space (eg. gaussian)</li>
  <li>parameterize <span class="math-container">$p(x | z)$</span> by the <em>decoder</em> network</li>
  <li>parameterize <span class="math-container">$q(z)$</span> by the <em>encoder</em> network (eg. <span class="math-container">$z \sim \mathcal{N}( \mu(x), \sigma(x))$</span>)</li>
  </ol>
  <p>Now <em><span class="math-container">$p(x|z)$</span> is not fixed, but rather fit</em>. So, as we train, we want to improve both the approximate inference model (encoder) and the generative model (decoder). This means that <span class="math-container">$p(x)$</span> is not fixed either as we train, so by increasing <span class="math-container">$L(x)$</span>, we can either increase <span class="math-container">$\log p(x)$</span> or decrease <span class="math-container">$D_{KL}\big(q(z) \big|\big| p(z|x) \big)$</span> or both.</p>
  <p>Why is increasing <span class="math-container">$\log p(x)$</span> a good thing? <span class="math-container">$\log p(x)$</span> tells us how likely that data we observe are under our model, this is a max likelihood fit of our (decoder parameterized) model. However, like the true posterior, <span class="math-container">$p(x)$</span> is difficult to calculate closed form, hence the ELBO.</p>
  <p>That's the key idea of the VAE: it expresses the model max likelihood objective (which is hard to compute) in terms of reconstruction loss (which is easy to compute), giving us a generative model and approximate posterior along the way.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-09T00:04:33.600Z
Answers.LastActivityDate: 2023-03-09T00:04:33.600Z
Answers.UserDisplayName: Luke
Answers.Body: <blockquote>
  <p>I thought <span class="math-container">$p(x)$</span> was just the distribution of <span class="math-container">$x$</span>, which are the input variables we have observed. So how can we maximize <span class="math-container">$p(x)$</span>, if it is the actual distribution of data in the real world?</p>
  </blockquote>
  <p>I think the main confusion is coming from what &quot;maximize&quot; means in all that. What you have is a family of functions <span class="math-container">$p_\theta(x)$</span>, parametrized by some parameters <span class="math-container">$\theta$</span>. It could be a neural network or some distribution in analytic form or some kind of <a href="https://en.wikipedia.org/wiki/Graphical_model" rel="nofollow noreferrer">graphical model</a>. By varying the parameters <span class="math-container">$\theta$</span> you are getting different distributions over <span class="math-container">$x$</span>.</p>
  <p>Now, you also have a sample of <span class="math-container">$x$</span>'es that you observed &quot;in the real world&quot;.  And we would like to find such parameters <span class="math-container">$\theta$</span> that <span class="math-container">$p_\theta(x)$</span> reflects the real world distribution as close as possible. The standard way to do that is to find <span class="math-container">$\theta$</span> that maximizes the log-likelihood of the observed data <span class="math-container">$x$</span>:
  <span class="math-container">$$\theta_{best} = \arg \max_\theta \log p_\theta(x)$$</span>
  This is called the <a href="https://en.wikipedia.org/wiki/Maximum_likelihood_estimation" rel="nofollow noreferrer">maximum likelihood estimation</a>. There are many extra details about it in the context of variational inference and VAEs, but that's the basic idea of what &quot;maximize&quot; means in all these cases.</p>
  <blockquote>
  <p>The Variational AutoEncoder is trying to learn an encoding from <span class="math-container">$x$</span> to <span class="math-container">$z$</span> and decoding from <span class="math-container">$z$</span> to <span class="math-container">$x'$</span>, such that the difference between <span class="math-container">$x$</span> and <span class="math-container">$x'$</span> is minimized.</p>
  </blockquote>
  <p>You've described a general case of autoencoder - &quot;variational&quot; adds some extra  details to it.</p>
  <p>The &quot;decoder&quot; in VAE is not just a function from  <span class="math-container">$z$</span> to <span class="math-container">$x$</span> - it is a probability distribution <span class="math-container">$p_\theta(x,z)$</span>. With <span class="math-container">$\theta$</span> being parameters of the decoding neural network. Given some values for latent variables <span class="math-container">$z$</span> you obtain a distribution over observed variables <span class="math-container">$x$</span>.</p>
  <p>For the &quot;encoder&quot; we would ideally like to just have a conditional probability distribution of <span class="math-container">$z$</span> given <span class="math-container">$x$</span>:
  <span class="math-container">$$p_\theta(z|x) = \frac{p_\theta(x,z)}{p_\theta(x)} = \frac{p_\theta(x,z)}{\int  p_\theta(x,z) dz}$$</span>
  The problem is in the denominator - the integral for marginal density of observations (also called <em>evidence</em>) <span class="math-container">$p_\theta(x)$</span> is computationally very hard to evaluate. This is what is called <a href="https://stats.stackexchange.com/a/343729/34983">intractability</a> of the evidence and the reason for the whole field of <a href="https://en.wikipedia.org/wiki/Variational_Bayesian_methods" rel="nofollow noreferrer">variational inference</a> to exist.</p>
  <p>VAE approximates the true posterior <span class="math-container">$p_\theta(z|x)$</span> with a variational approximation  <span class="math-container">$q_\phi(z|x)$</span>. Where <span class="math-container">$\phi$</span> are parameters of the encoder network, the distribution  usually factorizes following <a href="https://en.wikipedia.org/wiki/Variational_Bayesian_methods#Mean_field_approximation" rel="nofollow noreferrer">mean field approximation</a> approach.</p>
  <p>The need for <span class="math-container">$q_\phi(z|x)$</span> to approximate <span class="math-container">$p_\theta(z|x)$</span> leads to the whole ELBO derivation and the KL divergence term in the VAE loss.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-10T09:13:08.037Z
Answers.Text: Thank you for your helpful answer. 
  
  Does this mean that p(x) is the entire model? i.e. it is the function that includes both encoder and decoder and takes an x as input and as output produces another x', which could be different each time due to the sampling from the latent space? 
  
  And we are trying to make sure that p(x) over all the x in our dataset produces a distribution of x's that are as similar as possible to our original dataset?
Answers.UserDisplayName: Titus Buckworth
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-10T16:11:02.897Z
Answers.Text: @TitusBuckworth I would say that the "entire model" is $p_\theta(x, z)$. (Again, I encourage you to write the optimization parameters like $\theta$ explicitly.) You observe only $x$ and you want to simultaneously calculate a good maximum likelihood estimation of $\theta$ and infer the latent variables $z$, which you don't observe. This is a hard problem due to intractability of the integral for the evidence.  Variational approximation $q_\phi(z|x)$ is used to deal with this intractability.
Answers.UserDisplayName: Kostya
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-10T16:12:08.030Z
Answers.Text: @TitusBuckworth See this answer https://ai.stackexchange.com/a/27991/20538 for in-depth math on that.
Answers.UserDisplayName: Kostya
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-14T11:54:43.217Z
Answers.Text: Thanks. So you say that $p_\theta(x)$ is a family of functions. What is the input and output of those functions? What is the input and output for $p_\theta(x)$ with optimal $\theta$?
Answers.UserDisplayName: Titus Buckworth
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-14T12:13:51.213Z
Answers.Text: @TitusBuckworth It is a family of probability distributions. I'd really recommend reading Wikipedia link on Maximum Likelihood Estimation.
Answers.UserDisplayName: Kostya
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-09T12:06:04.333Z
Answers.LastActivityDate: 2023-03-09T12:06:04.333Z
Answers.UserDisplayName: Kostya
Body: <h3>Background</h3>
  <p>The loss function in a Variational AutoEncoder is the Evidence Lower Bound (ELBO):</p>
  <p><span class="math-container">$\mathbb{E}_q[log$</span> <span class="math-container">$p(x|z)] - KL[q(z)||p(z)]$</span></p>
  <p>And has this inequality:</p>
  <p><span class="math-container">$log$</span> <span class="math-container">$p(x) \ge \mathbb{E}_q[log$</span> <span class="math-container">$p(x|z)] - KL[q(z)||p(z)]$</span></p>
  <p>It is said in the literature that we want to maximise <span class="math-container">$log$</span> <span class="math-container">$p(x)$</span>, but instead, we maximise the ELBO, which we know is less than or equal to <span class="math-container">$log$</span> <span class="math-container">$p(x)$</span> (because this is an easier objective).</p>
  <h3>First Question</h3>
  <p>I am struggling to understand why the objective is to maximise <span class="math-container">$log$</span> <span class="math-container">$p(x)$</span> in the first place.</p>
  <p>Why is it not enough to maximise <span class="math-container">$\mathbb{E}_q[log$</span> <span class="math-container">$p(x|z)]$</span>, which is the reconstruction loss?</p>
  <p>I think I don't understand what exactly <span class="math-container">$p(x)$</span> means here.</p>
  <p>I thought <span class="math-container">$p(x)$</span> was just the distribution of <span class="math-container">$x$</span>, which are the input variables we have observed.<br />
  So how can we maximise <span class="math-container">$p(x)$</span>, if it is the actual distribution of data in the real world?</p>
  <h3>Main Question</h3>
  <p>The Variational AutoEncoder is trying to learn an encoding from <span class="math-container">$x$</span> to <span class="math-container">$z$</span> and decoding from <span class="math-container">$z$</span> to <span class="math-container">$x'$</span>, such that the difference between <span class="math-container">$x$</span> and <span class="math-container">$x'$</span> is minimized.</p>
  <p>How does <span class="math-container">$log$</span> <span class="math-container">$p(x)$</span> relate to this?<br />
  What does it mean and how does maximising it help?</p>
  
Tags: <variational-autoencoder><variational-inference><bayesian-inference>
Title: Why optimise log p(x) rather than log p(x|z) in a Variational AutoEncoder?
UserDisplayName: Titus Buckworth
ThreadText: # Why optimise log p(x) rather than log p(x|z) in a Variational AutoEncoder?
  
  (Asked by: username_0 on 2023-03-08)
  
  ### Background
  
  The loss function in a Variational AutoEncoder is the Evidence Lower Bound (ELBO):
  
  $\mathbb{E}\_q[log$ $p(x|z)] - KL[q(z)||p(z)]$
  
  And has this inequality:
  
  $log$ $p(x) \ge \mathbb{E}\_q[log$ $p(x|z)] - KL[q(z)||p(z)]$
  
  It is said in the literature that we want to maximise $log$ $p(x)$, but instead, we maximise the ELBO, which we know is less than or equal to $log$ $p(x)$ (because this is an easier objective).
  
  ### First Question
  
  I am struggling to understand why the objective is to maximise $log$ $p(x)$ in the first place.
  
  Why is it not enough to maximise $\mathbb{E}\_q[log$ $p(x|z)]$, which is the reconstruction loss?
  
  I think I don't understand what exactly $p(x)$ means here.
  
  I thought $p(x)$ was just the distribution of $x$, which are the input variables we have observed.  
  So how can we maximise $p(x)$, if it is the actual distribution of data in the real world?
  
  ### Main Question
  
  The Variational AutoEncoder is trying to learn an encoding from $x$ to $z$ and decoding from $z$ to $x'$, such that the difference between $x$ and $x'$ is minimized.
  
  How does $log$ $p(x)$ relate to this?  
  What does it mean and how does maximising it help?
  
  **Tags:** variational-autoencoder, variational-inference, bayesian-inference
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2023-03-09. Score: 1
  
  **TLDR:** We're doing a maximum likelihood fit of our model. The VAE sets this up in a way that doesn't require evaluating the model likelihood, but instead expresses a lower bound in terms of reconstruction error.
  
  Here's an explanation in two parts.
  
  #### 1. Variational inference in general
  
  We have a generative model $z \rightarrow x$ with latent variable $z \sim p(z)$, and observations, $x$, generated by $p(x|z)$. We want to get the posterior $p(z|x)$, but this inference may be computationally infeasible even if $p(z)$ and $p(x|z)$ are known *because $p(x)$ is hard to evaluate*. So, we instead approximate $p(z|x)$ by $q(z)$.
  
  Denoting the ELBO by $\mathcal{L}(x)$, we have (by expanding $D\_{KL}$):
  $$ \log p(x) = D\_{KL}\big(q(z) \big|\big| p(z|x) \big) + \mathcal{L}(x). $$
  Note that $p(x)$ is part of the generative model so, given data $x$, $p(x)$ is *constant* regardless of the approximation $q(z)$. This means that increasing $\mathcal{L}(x)$ by changing $q(z)$ is equivalent to decreasing $D\_{KL}\big(q(z) \big|\big| p(z|x) \big)$ and vice versa.
  
  Thus, the max ELBO approximation is the best in the sense that $q(z)$ is closest to $p(z|x)$ in KL terms. This is the classic reasoning for maximizing the ELBO.
  
  #### 2. Variational autoencoder
  
  For the variational autoencoder model, we
  
  1. choose $p(z)$ for the latent space (eg. gaussian)
  2. parameterize $p(x | z)$ by the *decoder* network
  3. parameterize $q(z)$ by the *encoder* network (eg. $z \sim \mathcal{N}( \mu(x), \sigma(x))$)
  
  Now *$p(x|z)$ is not fixed, but rather fit*. So, as we train, we want to improve both the approximate inference model (encoder) and the generative model (decoder). This means that $p(x)$ is not fixed either as we train, so by increasing $L(x)$, we can either increase $\log p(x)$ or decrease $D\_{KL}\big(q(z) \big|\big| p(z|x) \big)$ or both.
  
  Why is increasing $\log p(x)$ a good thing? $\log p(x)$ tells us how likely that data we observe are under our model, this is a max likelihood fit of our (decoder parameterized) model. However, like the true posterior, $p(x)$ is difficult to calculate closed form, hence the ELBO.
  
  That's the key idea of the VAE: it expresses the model max likelihood objective (which is hard to compute) in terms of reconstruction loss (which is easy to compute), giving us a generative model and approximate posterior along the way.
  
  
  ---
  
  ### Answer by username_2 on 2023-03-09. Score: 1
  
  > I thought $p(x)$ was just the distribution of $x$, which are the input variables we have observed. So how can we maximize $p(x)$, if it is the actual distribution of data in the real world?
  
  I think the main confusion is coming from what "maximize" means in all that. What you have is a family of functions $p\_\theta(x)$, parametrized by some parameters $\theta$. It could be a neural network or some distribution in analytic form or some kind of [graphical model](https://en.wikipedia.org/wiki/Graphical_model). By varying the parameters $\theta$ you are getting different distributions over $x$.
  
  Now, you also have a sample of $x$'es that you observed "in the real world". And we would like to find such parameters $\theta$ that $p\_\theta(x)$ reflects the real world distribution as close as possible. The standard way to do that is to find $\theta$ that maximizes the log-likelihood of the observed data $x$:
  $$\theta\_{best} = \arg \max\_\theta \log p\_\theta(x)$$
  This is called the [maximum likelihood estimation](https://en.wikipedia.org/wiki/Maximum_likelihood_estimation). There are many extra details about it in the context of variational inference and VAEs, but that's the basic idea of what "maximize" means in all these cases.
  
  > The Variational AutoEncoder is trying to learn an encoding from $x$ to $z$ and decoding from $z$ to $x'$, such that the difference between $x$ and $x'$ is minimized.
  
  You've described a general case of autoencoder - "variational" adds some extra details to it.
  
  The "decoder" in VAE is not just a function from $z$ to $x$ - it is a probability distribution $p\_\theta(x,z)$. With $\theta$ being parameters of the decoding neural network. Given some values for latent variables $z$ you obtain a distribution over observed variables $x$.
  
  For the "encoder" we would ideally like to just have a conditional probability distribution of $z$ given $x$:
  $$p\_\theta(z|x) = \frac{p\_\theta(x,z)}{p\_\theta(x)} = \frac{p\_\theta(x,z)}{\int p\_\theta(x,z) dz}$$
  The problem is in the denominator - the integral for marginal density of observations (also called *evidence*) $p\_\theta(x)$ is computationally very hard to evaluate. This is what is called [intractability](https://stats.stackexchange.com/a/343729/34983) of the evidence and the reason for the whole field of [variational inference](https://en.wikipedia.org/wiki/Variational_Bayesian_methods) to exist.
  
  VAE approximates the true posterior $p\_\theta(z|x)$ with a variational approximation $q\_\phi(z|x)$. Where $\phi$ are parameters of the encoder network, the distribution usually factorizes following [mean field approximation](https://en.wikipedia.org/wiki/Variational_Bayesian_methods#Mean_field_approximation) approach.
  
  The need for $q\_\phi(z|x)$ to approximate $p\_\theta(z|x)$ leads to the whole ELBO derivation and the KL divergence term in the VAE loss.
  
  > **Comments:**
  > **username_0**: Thank you for your helpful answer.   Does this mean that p(x) is the entire model? i.e. it is the function that includes both encoder and decoder and takes an x as input and as output produces another x', which could be different each time due to the sampling from the latent space?   And we are trying to make sure that p(x) over all the x in our dataset produces a distribution of x's that are as similar as possible to our original dataset?  
  > **username_2**: @username_0 I would say that the "entire model" is $p_\theta(x, z)$. (Again, I encourage you to write the optimization parameters like $\theta$ explicitly.) You observe only $x$ and you want to simultaneously calculate a good maximum likelihood estimation of $\theta$ and infer the latent variables $z$, which you don't observe. This is a hard problem due to intractability of the integral for the evidence.  Variational approximation $q_\phi(z|x)$ is used to deal with this intractability.  
  > **username_2**: @username_0 See this answer https://ai.stackexchange.com/a/27991/20538 for in-depth math on that.  
  > **username_0**: Thanks. So you say that $p_\theta(x)$ is a family of functions. What is the input and output of those functions? What is the input and output for $p_\theta(x)$ with optimal $\theta$?  
  > **username_2**: @username_0 It is a family of probability distributions. I'd really recommend reading Wikipedia link on Maximum Likelihood Estimation.  
  
  ---
  
  