Answers.Body: <p>In variational inference, the original objective is to minimize the Kullback-Leibler divergence between the variational distribution, <span class="math-container">$q(z \mid x)$</span>, and the posterior, <span class="math-container">$p(z \mid x) = \frac{p(x, z)}{\int_z p(x, z)}$</span>, given that the posterior can be difficult to directly infer with the Bayes rule, due to the denominator term, which can contain an intractable integral. </p>
  
  <p>Therefore, more formally, the optimization objective can be written as</p>
  
  <p><span class="math-container">\begin{align}
  q^*(z \mid x) = \operatorname{argmin}_{q(z \mid x)} D_{\text{KL}}(q(z \mid x) \| p(z \mid x))\tag{1} \label{1}
  \end{align}</span></p>
  
  <p>However, solving this optimization problem can be as difficult as the original inference one of computing the posterior <span class="math-container">$p(z \mid x)$</span> using the Bayes rule, given that it still involves the possibly intractable term <span class="math-container">$p(z \mid x)$</span>.</p>
  
  <p>If you use the definition of the KL divergence, you can derive the following equation</p>
  
  <p><span class="math-container">\begin{align}
  D_{\text{KL}}(q(z \mid x) \| p(z \mid x)) = \mathbb{E}_{q(z \mid x)} \left[ \log q(z \mid x) \right] - \mathbb{E}_{q(z \mid x)} \left[ \log q(z, x) \right] + \log p(x) \tag{2} \label{2}
  \end{align}</span></p>
  
  <p>First, note that the expectations are with respect to the variational distribution, which means that, if you want to approximate these expectations with Monte Carlo estimates, you can do it with respect to the variational distribution, and, given that it is assumed that one can easily sample from the variational distribution (which can e.g. be a Gaussian), this is a nice feature. </p>
  
  <p>Second, the KL divergence contains the term <span class="math-container">$p(x) = \int_z p(x, z)$</span>, the denominator term in the Bayes rule to compute the posterior <span class="math-container">$p(z \mid x)$</span>, which (as I said) can be intractable. <span class="math-container">$p(x)$</span> is often called the <strong>evidence</strong>.</p>
  
  <p>The solution is then to optimize an objective that does not contain this annoying intractable term <span class="math-container">$p(x)$</span>. The objective that is optimized is the so-called ELBO objective</p>
  
  <p><span class="math-container">\begin{align}
  \text{ELBO}(q) = \mathbb{E}_{q(z \mid x)} \left[ \log q(z, x) \right] - \mathbb{E}_{q(z \mid x)} \left[ \log q(z \mid x) \right]\tag{3} \label{3}
  \end{align}</span></p>
  
  <p>The KL divergence \ref{2} and the ELBO objective \ref{3} are similar. In fact, <strong>ELBO</strong> is an abbreviation for <strong>E</strong>vidence <strong>L</strong>ower <strong>BO</strong>und, because the ELBO is a lower bound on the evidence <span class="math-container">$p(x)$</span>, i.e. it is a number that is smaller than <span class="math-container">$p(x)$</span> or, more formally, <span class="math-container">$\text{ELBO}(q) \leq \log p(x)$</span>. Therefore, if we maximize <span class="math-container">$\text{ELBO}(q)$</span>, we also maximize the evidence <span class="math-container">$p(x)$</span> of the data (where <span class="math-container">$x$</span> is the data in your dataset). </p>
  
  <p>So, the objective in variational inference is</p>
  
  <p><span class="math-container">\begin{align}
  q^*(z \mid x) 
  &amp;= \operatorname{argmax}_{q(z \mid x)} \operatorname{ELBO}({q}) \\
  &amp;= \operatorname{argmax}_{q(z \mid x)}  \mathbb{E}_{q(z \mid x)} \left[ \log q(z, x) \right] - \mathbb{E}_{q(z \mid x)} \left[ \log q(z \mid x) \right]
  \tag{4} \label{4}
  \end{align}</span></p>
  
  <p>First, note that \ref{4} only contains terms that depend on the variational distribution, so we got rid of intractable terms, which was our goal.</p>
  
  <p>Second, note that, as opposed to \ref{1}, we are maximizing (or finding the parameters that maximize the objective).</p>
  
  <p>The ELBO objective is actually the <strong>negative</strong> of \ref{2} plus the logarithm of the evidence term, <span class="math-container">$\log p(x)$</span> (and you can easily verify it), that is</p>
  
  <p><span class="math-container">\begin{align}
  \text{ELBO}(q)  = -D_{\text{KL}}(q(z \mid x) \| p(z \mid x)) + \log p(x)
  \end{align}</span></p>
  
  <p>which can also be rearranged as</p>
  
  <p><span class="math-container">\begin{align}
  \log p(x) = D_{\text{KL}}(q(z \mid x) \| p(z \mid x)) + \text{ELBO}(q) \tag{5}\label{5}
  \end{align}</span></p>
  
  <p>which is your equation (where <span class="math-container">$\text{ELBO}(q)$</span> is your <span class="math-container">$\mathcal{L}$</span>). Therefore, your equation is true <strong>by definition</strong>, i.e. we define the ELBO such that \ref{5} is true. However, note that we haven't defined the ELBO in the way we dit only for the sake of it, but because it is a lower bound on the log evidence (and this follows from the fact that the KL divergence is never negative).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-07T11:45:58.920Z
Answers.LastActivityDate: 2020-02-09T14:11:28.613Z
Answers.LastEditDate: 2020-02-09T14:11:28.613Z
Answers.UserDisplayName: nbro
Body: <p>Why is the equation <span class="math-container">$$\log p_{\theta}(x^1,...,x^N)=D_{KL}(q_{\theta}(z|x^i)||p_{\phi}(z|x^i))+\mathbb{L}(\phi,\theta;x^i)$$</span> true, where <span class="math-container">$x^i$</span> are data points and <span class="math-container">$z$</span> are latent variables?</p>
  <p>I was reading the original variation autoencoder paper and I don't understand how the marginal is equal to the RHS equation. How does the marginal equal the KL divergence of <span class="math-container">$p$</span> with its approximate distribution plus the variational lower bound?</p>
  
Tags: <objective-functions><variational-autoencoder><latent-variable><kl-divergence><evidence-lower-bound>
Title: Why is the evidence equal to the KL divergence plus the loss?
UserDisplayName: user8714896
ThreadText: # Why is the evidence equal to the KL divergence plus the loss?
  
  (Asked by: username_0 on 2020-02-07)
  
  Why is the equation $$\log p\_{\theta}(x^1,...,x^N)=D\_{KL}(q\_{\theta}(z|x^i)||p\_{\phi}(z|x^i))+\mathbb{L}(\phi,\theta;x^i)$$ true, where $x^i$ are data points and $z$ are latent variables?
  
  I was reading the original variation autoencoder paper and I don't understand how the marginal is equal to the RHS equation. How does the marginal equal the KL divergence of $p$ with its approximate distribution plus the variational lower bound?
  
  **Tags:** objective-functions, variational-autoencoder, latent-variable, kl-divergence, evidence-lower-bound
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-02-07. Score: 8 | Accepted answer
  
  In variational inference, the original objective is to minimize the Kullback-Leibler divergence between the variational distribution, $q(z \mid x)$, and the posterior, $p(z \mid x) = \frac{p(x, z)}{\int\_z p(x, z)}$, given that the posterior can be difficult to directly infer with the Bayes rule, due to the denominator term, which can contain an intractable integral.
  
  Therefore, more formally, the optimization objective can be written as
  
  \begin{align}
  q^\*(z \mid x) = \operatorname{argmin}\_{q(z \mid x)} D\_{\text{KL}}(q(z \mid x) \| p(z \mid x))\tag{1} \label{1}
  \end{align}
  
  However, solving this optimization problem can be as difficult as the original inference one of computing the posterior $p(z \mid x)$ using the Bayes rule, given that it still involves the possibly intractable term $p(z \mid x)$.
  
  If you use the definition of the KL divergence, you can derive the following equation
  
  \begin{align}
  D\_{\text{KL}}(q(z \mid x) \| p(z \mid x)) = \mathbb{E}\_{q(z \mid x)} \left[ \log q(z \mid x) \right] - \mathbb{E}\_{q(z \mid x)} \left[ \log q(z, x) \right] + \log p(x) \tag{2} \label{2}
  \end{align}
  
  First, note that the expectations are with respect to the variational distribution, which means that, if you want to approximate these expectations with Monte Carlo estimates, you can do it with respect to the variational distribution, and, given that it is assumed that one can easily sample from the variational distribution (which can e.g. be a Gaussian), this is a nice feature.
  
  Second, the KL divergence contains the term $p(x) = \int\_z p(x, z)$, the denominator term in the Bayes rule to compute the posterior $p(z \mid x)$, which (as I said) can be intractable. $p(x)$ is often called the **evidence**.
  
  The solution is then to optimize an objective that does not contain this annoying intractable term $p(x)$. The objective that is optimized is the so-called ELBO objective
  
  \begin{align}
  \text{ELBO}(q) = \mathbb{E}\_{q(z \mid x)} \left[ \log q(z, x) \right] - \mathbb{E}\_{q(z \mid x)} \left[ \log q(z \mid x) \right]\tag{3} \label{3}
  \end{align}
  
  The KL divergence \ref{2} and the ELBO objective \ref{3} are similar. In fact, **ELBO** is an abbreviation for **E**vidence **L**ower **BO**und, because the ELBO is a lower bound on the evidence $p(x)$, i.e. it is a number that is smaller than $p(x)$ or, more formally, $\text{ELBO}(q) \leq \log p(x)$. Therefore, if we maximize $\text{ELBO}(q)$, we also maximize the evidence $p(x)$ of the data (where $x$ is the data in your dataset).
  
  So, the objective in variational inference is
  
  \begin{align}
  q^\*(z \mid x)
  &= \operatorname{argmax}\_{q(z \mid x)} \operatorname{ELBO}({q}) \\
  &= \operatorname{argmax}\_{q(z \mid x)} \mathbb{E}\_{q(z \mid x)} \left[ \log q(z, x) \right] - \mathbb{E}\_{q(z \mid x)} \left[ \log q(z \mid x) \right]
  \tag{4} \label{4}
  \end{align}
  
  First, note that \ref{4} only contains terms that depend on the variational distribution, so we got rid of intractable terms, which was our goal.
  
  Second, note that, as opposed to \ref{1}, we are maximizing (or finding the parameters that maximize the objective).
  
  The ELBO objective is actually the **negative** of \ref{2} plus the logarithm of the evidence term, $\log p(x)$ (and you can easily verify it), that is
  
  \begin{align}
  \text{ELBO}(q) = -D\_{\text{KL}}(q(z \mid x) \| p(z \mid x)) + \log p(x)
  \end{align}
  
  which can also be rearranged as
  
  \begin{align}
  \log p(x) = D\_{\text{KL}}(q(z \mid x) \| p(z \mid x)) + \text{ELBO}(q) \tag{5}\label{5}
  \end{align}
  
  which is your equation (where $\text{ELBO}(q)$ is your $\mathcal{L}$). Therefore, your equation is true **by definition**, i.e. we define the ELBO such that \ref{5} is true. However, note that we haven't defined the ELBO in the way we dit only for the sake of it, but because it is a lower bound on the log evidence (and this follows from the fact that the KL divergence is never negative).
  
  
  ---
  
  