Answers.Body: <p>Practically, when optimizing VAE, you assume that prior <span class="math-container">$p(z)\sim N(0,1)$</span>; i.e. the unit Gaussian distribution. However, in testime you sample z from <span class="math-container">$p(z|x)$</span>; the encoder model. Why is that?</p>
  <p>Let's go back to the start. We have a model <span class="math-container">$p_{\theta}(x)$</span> and the data <span class="math-container">$\{x_1, ..., x_N\}$</span>. Solving the maximum log-likelihood problem, we have
  <span class="math-container">\begin{equation}
  \begin{split}
  \theta &amp;= argmax_{\theta}\frac{1}{N}\sum_i log p_{\theta}(x_i) \\
  &amp;= argmax_{\theta}\frac{1}{N}\sum_i \left( \int p_{\theta}(x_i|z)p(z)dz \right)
  \end{split}
  \end{equation}</span></p>
  <p>which is intractable to calculate. So what to do now?</p>
  <p>Here it comes the <strong>Variational Inference</strong>: &quot;Use the expected log-likelihood instead.&quot;
  <span class="math-container">\begin{equation}
  \begin{split}
  \theta &amp;= argmax_{\theta}\sum_i E_{z \sim p_{\theta}(z|x)}\left[ logp_{\theta}(x,z) \right]
  \end{split}
  \end{equation}</span></p>
  <p>We approximate <span class="math-container">$q(z) \approx p_{\theta}(z|x)$</span>. Thus, we unfold <span class="math-container">$logp(x)$</span>:
  <span class="math-container">\begin{equation}
  \begin{split}
  logp(x) &amp;= log \int p_{\theta}(x_i|z)p(z)dz \\
  &amp;= log \int p_{\theta}(x_i|z)p(z) \frac{q(z)}{q(z)}dz \\
  &amp;= log E_{z \sim q(z)}\left[ \frac{p_{\theta}(x|z)p(z)}{q(z)} \right] \\
  &amp;\geq E_{z \sim q(z)}\left[ log \frac{p_{\theta}(x|z)p(z)}{q(z)} \right]  \\
  &amp;= E_{z \sim q(z)}\left[ logp_{\theta}(x|z) + logp(z) \right] - E_{z \sim q(z)}\left[ logq(z)\right] \\
  &amp;= E_{z \sim q(z)}\left[ logp_{\theta}(x|z) + logp(z) \right] + H(q) \\
  &amp;= ELBO(p,q)
  \end{split}
  \end{equation}</span></p>
  <p>where <span class="math-container">$H(q)$</span> is the entropy of q. As you highlighted we can also write
  <span class="math-container">$$ ELBO(p,q) = logp(x) - D_{KL}\left( q(z) || p(z|x) \right) $$</span></p>
  <p>This means that (a) maximizing <span class="math-container">$ELBO(p,q)$</span> w.r.t. to <span class="math-container">$q$</span> then KL-Divergence is minimized and (b) maximizing <span class="math-container">$ELBO(p,q)$</span> w.r.t to <span class="math-container">$p$</span> then the model is improved as the log-likelihood is improved. So this point of yours is true.</p>
  <p><strong>However, how can we actually train this model?</strong></p>
  <p>The answer is by using <strong>Amortized Variational Inference</strong>! Practically, we use 2 Neural Networks, <span class="math-container">$\phi$</span> and <span class="math-container">$\theta$</span>, so that we have 2 models: <span class="math-container">$q_{\phi}(z|x)$</span> (encoder) and <span class="math-container">$p_{\theta}(x|z)$</span> (decoder). Thus, we replace <span class="math-container">$q(z)$</span> with <span class="math-container">$q_{\phi}(z|x)$</span> and <span class="math-container">$ELBO(p,q)$</span> with <span class="math-container">$ELBO(\theta,\phi)$</span> .</p>
  <p>To fully answer your question, I should reach your first ELBO formula. I will unfold my first equation about ELBO:</p>
  <p><span class="math-container">\begin{equation}
  \begin{split}
  ELBO(\theta,\phi) &amp;= E_{z \sim q_{\phi}(z)}\left[ logp_{\theta}(x|z)\right] + E_{z \sim q_{\phi}(z)}\left[ logp(z) \right] + H\left(q_{\phi}(z|x)\right) \\
  &amp;= E_{z \sim q_{\phi}(z)}\left[ logp_{\theta}(x|z)\right] - D_{KL}\left( q_{\phi}(z|x) || p_{\theta}(z) \right) 
  \end{split}
  \end{equation}</span></p>
  <p>Therefore, using the Amortized Variational Inference, we maximize the above objective (which is the same as the one above).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-22T14:23:24.047Z
Answers.Text: Amortized VI doesnt change the objective you use, they both hold, ultimately the aim of ELBO is to act as a lower bound for for the log likelihood, the proof for the first objective you show still holds if we use q(z|x) instead of q(z) meaning both objectives apply in amortized VI.
Answers.UserDisplayName: quest ions
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-22T23:30:38.913Z
Answers.Text: Of course. I clarified it on text above. Thanks for the feedback. I hope I answered your question above.
Answers.UserDisplayName: ddaedalus
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-24T19:24:17.960Z
Answers.Text: My question is what does the distribution q(z|x) tend to when training VAEs, i'm not sure how this answer addresses this.
Answers.UserDisplayName: quest ions
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-27T23:53:50.427Z
Answers.LastActivityDate: 2021-06-22T23:29:11.647Z
Answers.LastEditDate: 2021-06-22T23:29:11.647Z
Answers.UserDisplayName: ddaedalus
Body: <p>The ELBO objective is described as follows</p>
  <p><span class="math-container">$$ ELBO(\phi,\theta) = E_{q_\phi(z|x)}[log p_\theta (x|z)] - KL[q_\phi (z|x)||p(z)] $$</span></p>
  <p>This form of ELBO includes a regularisation term  in the form of the KL divergence which drives <span class="math-container">$q_\phi(z|x) \rightarrow p(z)$</span> when optimising ELBO.</p>
  <p>However we also have the overall expression for the loglikelihood which is defined as follows (<a href="https://arxiv.org/pdf/1906.02691.pdf#page=21" rel="nofollow noreferrer">proof provided here</a>)</p>
  <p><span class="math-container">$$ p_\theta(x) = ELBO(\phi,\theta) + KL[q_\phi(z|x)||p_\theta(z|x)] $$</span></p>
  <p>Rearranging the above equation as follows</p>
  <p><span class="math-container">$$  \max\limits_\phi ELBO(\phi,\theta) = \max\limits_\phi p_\theta(x) - KL[q_\phi(z|x)||p_\theta(z|x)]  $$</span></p>
  <p>We can see that maximising ELBO w.r.t <span class="math-container">$\phi$</span> in this form causes <span class="math-container">$q_\phi(z|x) \rightarrow p_\theta(z|x)$</span></p>
  <p>These two ways of describing how VAEs learn conflicts my understanding of what happens to the approximate distribution during training.</p>
  <p>Is it simply just trying to match both the prior <span class="math-container">$p(z)$</span> and the posterior <span class="math-container">$p_\theta(z|x)$</span> or am I missing something</p>
  
Tags: <variational-autoencoder><evidence-lower-bound><variational-inference>
Title: What does the approximate posterior on latent variables, $q_\phi(z|x)$, tend to when optimising VAE's
UserDisplayName: quest ions
ThreadText: # What does the approximate posterior on latent variables, $q_\phi(z|x)$, tend to when optimising VAE's
  
  (Asked by: username_0 on 2021-04-23)
  
  The ELBO objective is described as follows
  
  $$ ELBO(\phi,\theta) = E\_{q\_\phi(z|x)}[log p\_\theta (x|z)] - KL[q\_\phi (z|x)||p(z)] $$
  
  This form of ELBO includes a regularisation term in the form of the KL divergence which drives $q\_\phi(z|x) \rightarrow p(z)$ when optimising ELBO.
  
  However we also have the overall expression for the loglikelihood which is defined as follows ([proof provided here](https://arxiv.org/pdf/1906.02691.pdf#page=21))
  
  $$ p\_\theta(x) = ELBO(\phi,\theta) + KL[q\_\phi(z|x)||p\_\theta(z|x)] $$
  
  Rearranging the above equation as follows
  
  $$ \max\limits\_\phi ELBO(\phi,\theta) = \max\limits\_\phi p\_\theta(x) - KL[q\_\phi(z|x)||p\_\theta(z|x)] $$
  
  We can see that maximising ELBO w.r.t $\phi$ in this form causes $q\_\phi(z|x) \rightarrow p\_\theta(z|x)$
  
  These two ways of describing how VAEs learn conflicts my understanding of what happens to the approximate distribution during training.
  
  Is it simply just trying to match both the prior $p(z)$ and the posterior $p\_\theta(z|x)$ or am I missing something
  
  **Tags:** variational-autoencoder, evidence-lower-bound, variational-inference
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-05-27. Score: 1
  
  Practically, when optimizing VAE, you assume that prior $p(z)\sim N(0,1)$; i.e. the unit Gaussian distribution. However, in testime you sample z from $p(z|x)$; the encoder model. Why is that?
  
  Let's go back to the start. We have a model $p\_{\theta}(x)$ and the data $\{x\_1, ..., x\_N\}$. Solving the maximum log-likelihood problem, we have
  \begin{equation}
  \begin{split}
  \theta &= argmax\_{\theta}\frac{1}{N}\sum\_i log p\_{\theta}(x\_i) \\
  &= argmax\_{\theta}\frac{1}{N}\sum\_i \left( \int p\_{\theta}(x\_i|z)p(z)dz \right)
  \end{split}
  \end{equation}
  
  which is intractable to calculate. So what to do now?
  
  Here it comes the **Variational Inference**: "Use the expected log-likelihood instead."
  \begin{equation}
  \begin{split}
  \theta &= argmax\_{\theta}\sum\_i E\_{z \sim p\_{\theta}(z|x)}\left[ logp\_{\theta}(x,z) \right]
  \end{split}
  \end{equation}
  
  We approximate $q(z) \approx p\_{\theta}(z|x)$. Thus, we unfold $logp(x)$:
  \begin{equation}
  \begin{split}
  logp(x) &= log \int p\_{\theta}(x\_i|z)p(z)dz \\
  &= log \int p\_{\theta}(x\_i|z)p(z) \frac{q(z)}{q(z)}dz \\
  &= log E\_{z \sim q(z)}\left[ \frac{p\_{\theta}(x|z)p(z)}{q(z)} \right] \\
  &\geq E\_{z \sim q(z)}\left[ log \frac{p\_{\theta}(x|z)p(z)}{q(z)} \right] \\
  &= E\_{z \sim q(z)}\left[ logp\_{\theta}(x|z) + logp(z) \right] - E\_{z \sim q(z)}\left[ logq(z)\right] \\
  &= E\_{z \sim q(z)}\left[ logp\_{\theta}(x|z) + logp(z) \right] + H(q) \\
  &= ELBO(p,q)
  \end{split}
  \end{equation}
  
  where $H(q)$ is the entropy of q. As you highlighted we can also write
  $$ ELBO(p,q) = logp(x) - D\_{KL}\left( q(z) || p(z|x) \right) $$
  
  This means that (a) maximizing $ELBO(p,q)$ w.r.t. to $q$ then KL-Divergence is minimized and (b) maximizing $ELBO(p,q)$ w.r.t to $p$ then the model is improved as the log-likelihood is improved. So this point of yours is true.
  
  **However, how can we actually train this model?**
  
  The answer is by using **Amortized Variational Inference**! Practically, we use 2 Neural Networks, $\phi$ and $\theta$, so that we have 2 models: $q\_{\phi}(z|x)$ (encoder) and $p\_{\theta}(x|z)$ (decoder). Thus, we replace $q(z)$ with $q\_{\phi}(z|x)$ and $ELBO(p,q)$ with $ELBO(\theta,\phi)$ .
  
  To fully answer your question, I should reach your first ELBO formula. I will unfold my first equation about ELBO:
  
  \begin{equation}
  \begin{split}
  ELBO(\theta,\phi) &= E\_{z \sim q\_{\phi}(z)}\left[ logp\_{\theta}(x|z)\right] + E\_{z \sim q\_{\phi}(z)}\left[ logp(z) \right] + H\left(q\_{\phi}(z|x)\right) \\
  &= E\_{z \sim q\_{\phi}(z)}\left[ logp\_{\theta}(x|z)\right] - D\_{KL}\left( q\_{\phi}(z|x) || p\_{\theta}(z) \right)
  \end{split}
  \end{equation}
  
  Therefore, using the Amortized Variational Inference, we maximize the above objective (which is the same as the one above).
  
  > **Comments:**
  > **username_0**: Amortized VI doesnt change the objective you use, they both hold, ultimately the aim of ELBO is to act as a lower bound for for the log likelihood, the proof for the first objective you show still holds if we use q(z|x) instead of q(z) meaning both objectives apply in amortized VI.  
  > **username_1**: Of course. I clarified it on text above. Thanks for the feedback. I hope I answered your question above.  
  > **username_0**: My question is what does the distribution q(z|x) tend to when training VAEs, i'm not sure how this answer addresses this.  
  
  ---
  
  