Answers.Body: <p>From <a href="http://users.umiacs.umd.edu/~xyang35/files/understanding-variational-lower.pdf" rel="nofollow noreferrer">this document</a>, as you found here, <span class="math-container">$X$</span> is an observed variable and <span class="math-container">$Z$</span> is a hidden variable; <span class="math-container">$p(X)$</span> is the density function of <span class="math-container">$X$</span>. The posterior distribution of the hidden variables can then be written as follows using the Bayes’ Theorem:</p>
  
  <p><span class="math-container">$$p(Z|X) = \frac{p(X|Z)p(Z)}{p(X)} = \frac{p(X|Z)p(Z)}{\int_Zp(X,Z)}$$</span></p>
  
  <p>Now base on what you post, if we denote that <span class="math-container">$L= \mathbb{E}_q [\log p(X, Z)] + H[Z]$</span> (<span class="math-container">$q(Z)$</span> is a distribution we use to approximation the true posterior distribution <span class="math-container">$p(Z|X)$</span> in VB and <span class="math-container">$H[Z] = -\mathbb{E}_q [\log q(Z)]$</span>), then it is obvious that <span class="math-container">$L$</span> is a lower bound of the log probability of the observations.
  As a result, if in some cases we want to maximize the marginal probability (the log probability of the observations), we can instead maximize its variational lower bound <span class="math-container">$L$</span>. As a real example, you can follow the "Multiple Object Recognition with Visual Attention" example in the referenced document.</p>
  
  <p>Moreover, the term <span class="math-container">$L$</span> will be presented in KL-divergence that will be used to measuring the similarity of two distributions. Be aware that there is progress on the bound in <a href="http://proceedings.mlr.press/v80/alemi18a/alemi18a.pdf" rel="nofollow noreferrer">this paper</a> (Fixing a Broken ELBO).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-21T12:45:45.447Z
Answers.Text: You do not explain how $q(Z)$ is related to the last term. What is $H[Z]$? You also do not explain why we would want to maximize $p(X)$.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-21T13:00:43.037Z
Answers.Text: @nbro well noted. It is updated.
Answers.UserDisplayName: OmG
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-21T13:12:39.400Z
Answers.Text: Your answer seems correct, but I think it will still be difficult for a newbie to understand the ELBO and its motivation. Maybe later I will add a more complete answer.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-21T19:30:03.900Z
Answers.Text: what does log of the P(X) get us? Why do they log it?
Answers.UserDisplayName: Gooby
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-21T19:56:52.297Z
Answers.Text: @DylanY It is a marginal probability. Indeed, it is used in the concept of KL-Divergence. There, it is related to the concept of entropy that there is a $\log$ in the definition of the entropy.
Answers.UserDisplayName: OmG
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-21T19:57:41.313Z
Answers.Text: do you have any good learning resources on Entropy? I have incomplete idea of what it is and how it helps get the marginal.
Answers.UserDisplayName: Gooby
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-21T19:59:17.830Z
Answers.Text: @DylanY I think this link of wikipedia https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence could be helpful.
Answers.UserDisplayName: OmG
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-21T11:49:09.993Z
Answers.LastActivityDate: 2019-10-21T13:00:23.927Z
Answers.LastEditDate: 2019-10-21T13:00:23.927Z
Answers.UserDisplayName: OmG
Answers.Body: <p>The use of KL provides a more intuitive way of what the ELBO is attempting to maximize.</p>
  
  <p>Basically, we want to find a posterior approximation such that <span class="math-container">$p(z\mid x) \approx q(z)\in\mathcal{Q}$</span></p>
  
  <p><span class="math-container">$$KL(q(z)\parallel p(z\mid x)) \rightarrow \min_{q(z)\in\mathcal{Q}}$$</span></p>
  
  <p>As a result of this, while finding this optimal posterior approximation, we maximize the probability of all the observed data <span class="math-container">$x$</span>. Note that the evidence is usually intractable. Thus, can express the <span class="math-container">$KL$</span> as follows:</p>
  
  <p><span class="math-container">\begin{align*} \log p(x) &amp;= \int q(z) \log p(x)dz \\
  &amp;= \int q(z) \log\frac{p(x,\theta)}{p(\theta\mid x)}dz \\ 
  &amp;= \int q(z) \log\frac{p(x,z)q(z)}{p(\theta\mid x)q(z)}dz\\ 
  &amp;= \int q(z) \log\frac{p(x,z)}{q(z)}dz + \int q(z) \log\frac{q(z)}{p(z\mid x)}dz \\ &amp;= \mathcal{L}(q(z)) + KL(q(z)\parallel p(z\mid x)) \end{align*}</span></p>
  
  <p>In this case, KL just gives us the difference between <span class="math-container">$q$</span> and <span class="math-container">$p$</span>. We want to make this difference close to zero meaning that <span class="math-container">$q=p$</span>. So, minimizing the KL is the same as maximizing the ELBO, and as a result, we obtain the lower bound in your expression. If you expand your bound, you can find a nice interpretion: </p>
  
  <p><span class="math-container">$$ \begin{align*} 
  \mathcal{L}(q(z)) &amp;= \int q(z) \log\frac{p(x,z)}{q(z)}dz \\
  &amp;= \mathbb{E}_{q(z)} \log p(x\mid z) - KL(q(z)\parallel p(z)) \end{align*} $$</span></p>
  
  <p>When we optimize this expression, we want to find a <span class="math-container">$q$</span> that fits our data properly and also is really close to true posterior. Thus, <span class="math-container">$\mathbb{E}_{q(z)} \log p(x\mid z)$</span> act as a data term and <span class="math-container">$KL(q(z)\parallel p(z)) $</span> as a regularizer. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-21T15:13:02.053Z
Answers.Text: I think that the best way to motivate the ELBO is to start from the minimization problem of the KL divergence (which is the actual problem you want to solve), that is, the problem of minimizing the difference between the variational distribution (a supposedly tractable distribution) and the posterior (that you want to approximate with the variational distribution), then derive the ELBO. The motivation should not be that you want to maximize $p(x)$.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-21T14:54:31.187Z
Answers.LastActivityDate: 2019-11-03T14:50:42.060Z
Answers.LastEditDate: 2019-11-03T14:50:42.060Z
Answers.UserDisplayName: c.uent
Body: <p><a href="https://i.sstatic.net/bMmxd.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/bMmxd.png" alt="enter image description here" /></a></p>
  <p>I don't really understand what this equation is saying or what the purpose of the ELBO is. How does it help us find the true posterior distribution?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-10-21T11:34:53.953Z
Comments.Text: Can you specify the source of the image?
Comments.UserDisplayName: nbro
Tags: <deep-learning><statistical-ai><variational-autoencoder><evidence-lower-bound>
Title: What's going on in the equation of the variational lower bound?
UserDisplayName: Gooby
ThreadText: # What's going on in the equation of the variational lower bound?
  
  (Asked by: username_0 on 2019-10-21)
  
  [![enter image description here](https://i.sstatic.net/bMmxd.png)](https://i.sstatic.net/bMmxd.png)
  
  I don't really understand what this equation is saying or what the purpose of the ELBO is. How does it help us find the true posterior distribution?
  
  **Tags:** deep-learning, statistical-ai, variational-autoencoder, evidence-lower-bound
  
  > **Comments:**
  > **username_1**: Can you specify the source of the image?  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2019-10-21. Score: 3
  
  From [this document](http://users.umiacs.umd.edu/~xyang35/files/understanding-variational-lower.pdf), as you found here, $X$ is an observed variable and $Z$ is a hidden variable; $p(X)$ is the density function of $X$. The posterior distribution of the hidden variables can then be written as follows using the Bayes’ Theorem:
  
  $$p(Z|X) = \frac{p(X|Z)p(Z)}{p(X)} = \frac{p(X|Z)p(Z)}{\int\_Zp(X,Z)}$$
  
  Now base on what you post, if we denote that $L= \mathbb{E}\_q [\log p(X, Z)] + H[Z]$ ($q(Z)$ is a distribution we use to approximation the true posterior distribution $p(Z|X)$ in VB and $H[Z] = -\mathbb{E}\_q [\log q(Z)]$), then it is obvious that $L$ is a lower bound of the log probability of the observations.
  As a result, if in some cases we want to maximize the marginal probability (the log probability of the observations), we can instead maximize its variational lower bound $L$. As a real example, you can follow the "Multiple Object Recognition with Visual Attention" example in the referenced document.
  
  Moreover, the term $L$ will be presented in KL-divergence that will be used to measuring the similarity of two distributions. Be aware that there is progress on the bound in [this paper](http://proceedings.mlr.press/v80/alemi18a/alemi18a.pdf) (Fixing a Broken ELBO).
  
  > **Comments:**
  > **username_1**: You do not explain how $q(Z)$ is related to the last term. What is $H[Z]$? You also do not explain why we would want to maximize $p(X)$.  
  > **username_2**: @username_1 well noted. It is updated.  
  > **username_1**: Your answer seems correct, but I think it will still be difficult for a newbie to understand the ELBO and its motivation. Maybe later I will add a more complete answer.  
  > **username_0**: what does log of the P(X) get us? Why do they log it?  
  > **username_2**: @DylanY It is a marginal probability. Indeed, it is used in the concept of KL-Divergence. There, it is related to the concept of entropy that there is a $\log$ in the definition of the entropy.  
  > **username_0**: do you have any good learning resources on Entropy? I have incomplete idea of what it is and how it helps get the marginal.  
  > **username_2**: @DylanY I think this link of wikipedia https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence could be helpful.  
  
  ---
  
  ### Answer by username_3 on 2019-10-21. Score: 2
  
  The use of KL provides a more intuitive way of what the ELBO is attempting to maximize.
  
  Basically, we want to find a posterior approximation such that $p(z\mid x) \approx q(z)\in\mathcal{Q}$
  
  $$KL(q(z)\parallel p(z\mid x)) \rightarrow \min\_{q(z)\in\mathcal{Q}}$$
  
  As a result of this, while finding this optimal posterior approximation, we maximize the probability of all the observed data $x$. Note that the evidence is usually intractable. Thus, can express the $KL$ as follows:
  
  \begin{align\*} \log p(x) &= \int q(z) \log p(x)dz \\
  &= \int q(z) \log\frac{p(x,\theta)}{p(\theta\mid x)}dz \\
  &= \int q(z) \log\frac{p(x,z)q(z)}{p(\theta\mid x)q(z)}dz\\
  &= \int q(z) \log\frac{p(x,z)}{q(z)}dz + \int q(z) \log\frac{q(z)}{p(z\mid x)}dz \\ &= \mathcal{L}(q(z)) + KL(q(z)\parallel p(z\mid x)) \end{align\*}
  
  In this case, KL just gives us the difference between $q$ and $p$. We want to make this difference close to zero meaning that $q=p$. So, minimizing the KL is the same as maximizing the ELBO, and as a result, we obtain the lower bound in your expression. If you expand your bound, you can find a nice interpretion:
  
  $$ \begin{align\*}
  \mathcal{L}(q(z)) &= \int q(z) \log\frac{p(x,z)}{q(z)}dz \\
  &= \mathbb{E}\_{q(z)} \log p(x\mid z) - KL(q(z)\parallel p(z)) \end{align\*} $$
  
  When we optimize this expression, we want to find a $q$ that fits our data properly and also is really close to true posterior. Thus, $\mathbb{E}\_{q(z)} \log p(x\mid z)$ act as a data term and $KL(q(z)\parallel p(z)) $ as a regularizer.
  
  > **Comments:**
  > **username_1**: I think that the best way to motivate the ELBO is to start from the minimization problem of the KL divergence (which is the actual problem you want to solve), that is, the problem of minimizing the difference between the variational distribution (a supposedly tractable distribution) and the posterior (that you want to approximate with the variational distribution), then derive the ELBO. The motivation should not be that you want to maximize $p(x)$.  
  
  ---
  
  