Answers.Body: <p>Your description of what is going on is more or less correct, although I am not completely sure that you have really understood it, given your last question.</p>
  <p>So, let me enumerate the steps.</p>
  <ol>
  <li><p>The computation of the posterior is often <strong>intractable</strong> (given that the evidence, i.e. the denominator of the right-hand side of the Bayes' rule, might be numerically expensive to approximate/compute or there's no closed-form solution)</p>
  </li>
  <li><p>To address this intractability, you cast the Bayesian inference problem (i.e. the application of Bayes' rule) as an optimization problem</p>
  <ol>
  <li><p>You assume that you can approximate the posterior with another simpler distribution (e.g. a Gaussian), known as the <strong>variational distribution</strong></p>
  </li>
  <li><p>You <strong>formulate</strong> this optimization problem as the minimization of some notion of distance (e.g. the KL divergence) between the posterior and the VD</p>
  </li>
  <li><p>However, the KL divergence between the posterior and the VD turns out to be intractable too, given that, if you expand it, you will find out that there's still an evidence term</p>
  </li>
  <li><p>Therefore, you use a tractable <strong>surrogate</strong> (i.e. equivalent, up to some constant) objective function, which is known as the <strong>evidence lower bound (ELBO)</strong> (which is sometimes known as the <strong>variational free energy</strong>), which is the sum of 2 terms</p>
  <ol>
  <li>KL divergence between the VD and the prior</li>
  <li>the likelihood of the parameters given the data</li>
  </ol>
  </li>
  </ol>
  </li>
  </ol>
  <p>To address your last doubt/question, the ELBO does not contain the posterior (i.e. what you really want to find), but only the variational distribution (you choose this!), the prior (which you also define/choose), and the likelihood (which, in practice, corresponds to the typical usage of the cross-entropy; so the only thing that you need more, with respect to the traditional neural networks, is the computation of the KL divergence): in other words, you originally formulate the problem as the minimization of the KL divergence between the posterior and the VD, but this is just a formulation.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-22T11:16:48.780Z
Answers.Text: thanks! So in fact instead of using a point estimator or the fully bayesian approach, you still want to have a distribution (and not point estimates) but instead of applying the true bayesian inference, you formulate the task as an optimization. The intuitive explanation of the free energy term might then be s.th. like: we of course want to increase the likelihood of the parameters given the data but at the same time want to stay close to the priors? Optimizing this term gives us a good approx. of the true posterior
Answers.UserDisplayName: f_3464gh
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-22T11:18:49.733Z
Answers.Text: @manu Yes, exactly, I would say that's the typical intuition interpretation of what is going on when you optimize the ELBO.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-22T11:25:41.160Z
Answers.Text: Already, thanks so much! Just one more question - you said that the ELBO is equivalent to the KL up to some constant. What exactly is the constant? or does the marginal likelihood that becomes by calculating derivative refers to this constant?
Answers.UserDisplayName: f_3464gh
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-22T11:29:05.490Z
Answers.Text: The constant is the (log of the) evidence, as you had said in the original version of the post. The ELBO is **equal to** (the **negative** of the) "KL divergence between the posterior and the VD" + "the evidence": that's why **maximizing** the ELBO is equivalent to the original optimization problem of **minimizing** the "KL divergence between the posterior and the VD". As you also had said, the evidence is a constant with respect to the **variational parameters**, i.e. the mean and variance of the VD, which is what you want to find.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-22T11:09:46.890Z
Answers.LastActivityDate: 2021-03-22T14:05:47.067Z
Answers.LastEditDate: 2021-03-22T14:05:47.067Z
Answers.UserDisplayName: nbro
Body: <p>I'm trying to understand the concept of Variational Inference for BNNs. My source is this <a href="https://arxiv.org/pdf/2006.12024.pdf" rel="nofollow noreferrer">work</a>. The aim is to minimize the divergence between the approx. distribution and the true posterior</p>
  <p><span class="math-container">$$\text{KL}(q_{\theta}(w)||p(w|D) = \int q_{\theta}(w) \ log \frac{q_{\theta}(w)}{p(w \mid D)} \ dw$$</span></p>
  <p>This can be expanded out as <span class="math-container">$$- F[q_{\theta}] + \log \ p(D)$$</span> where <span class="math-container">$$F[q_{\theta}] = -\text{KL}(q_{\theta}(w) || p(w)) + E[\log p(D \mid w)]$$</span></p>
  <p>Because <span class="math-container">$log \ p(D)$</span> does not contain any variational parameters, the derivative will be zero. I really would like to summarize the concept of VI in words.</p>
  <p>How can one explain the last formula in words intuitive and with it on the fact that one approximates a function without really knowing it / able to compute it?</p>
  <p>My attempt would be: Minimizing the KL between the approximate distribution and the true posterior boils down in minimizing the KL between the approximate distribution and the prior (?) and maximizing the log-likelihood that the parameters of the approximate distribution resulted in the data. Is this somehow correct?</p>
  
Tags: <neural-networks><machine-learning><bayesian-deep-learning><bayesian-neural-networks><variational-inference>
Title: What is the intuition behind variational inference for Bayesian neural networks?
UserDisplayName: f_3464gh
ThreadText: # What is the intuition behind variational inference for Bayesian neural networks?
  
  (Asked by: username_0 on 2021-03-22)
  
  I'm trying to understand the concept of Variational Inference for BNNs. My source is this [work](https://arxiv.org/pdf/2006.12024.pdf). The aim is to minimize the divergence between the approx. distribution and the true posterior
  
  $$\text{KL}(q\_{\theta}(w)||p(w|D) = \int q\_{\theta}(w) \ log \frac{q\_{\theta}(w)}{p(w \mid D)} \ dw$$
  
  This can be expanded out as $$- F[q\_{\theta}] + \log \ p(D)$$ where $$F[q\_{\theta}] = -\text{KL}(q\_{\theta}(w) || p(w)) + E[\log p(D \mid w)]$$
  
  Because $log \ p(D)$ does not contain any variational parameters, the derivative will be zero. I really would like to summarize the concept of VI in words.
  
  How can one explain the last formula in words intuitive and with it on the fact that one approximates a function without really knowing it / able to compute it?
  
  My attempt would be: Minimizing the KL between the approximate distribution and the true posterior boils down in minimizing the KL between the approximate distribution and the prior (?) and maximizing the log-likelihood that the parameters of the approximate distribution resulted in the data. Is this somehow correct?
  
  **Tags:** neural-networks, machine-learning, bayesian-deep-learning, bayesian-neural-networks, variational-inference
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-03-22. Score: 9 | Accepted answer
  
  Your description of what is going on is more or less correct, although I am not completely sure that you have really understood it, given your last question.
  
  So, let me enumerate the steps.
  
  1. The computation of the posterior is often **intractable** (given that the evidence, i.e. the denominator of the right-hand side of the Bayes' rule, might be numerically expensive to approximate/compute or there's no closed-form solution)
  2. To address this intractability, you cast the Bayesian inference problem (i.e. the application of Bayes' rule) as an optimization problem
  
     1. You assume that you can approximate the posterior with another simpler distribution (e.g. a Gaussian), known as the **variational distribution**
     2. You **formulate** this optimization problem as the minimization of some notion of distance (e.g. the KL divergence) between the posterior and the VD
     3. However, the KL divergence between the posterior and the VD turns out to be intractable too, given that, if you expand it, you will find out that there's still an evidence term
     4. Therefore, you use a tractable **surrogate** (i.e. equivalent, up to some constant) objective function, which is known as the **evidence lower bound (ELBO)** (which is sometimes known as the **variational free energy**), which is the sum of 2 terms
  
        1. KL divergence between the VD and the prior
        2. the likelihood of the parameters given the data
  
  To address your last doubt/question, the ELBO does not contain the posterior (i.e. what you really want to find), but only the variational distribution (you choose this!), the prior (which you also define/choose), and the likelihood (which, in practice, corresponds to the typical usage of the cross-entropy; so the only thing that you need more, with respect to the traditional neural networks, is the computation of the KL divergence): in other words, you originally formulate the problem as the minimization of the KL divergence between the posterior and the VD, but this is just a formulation.
  
  > **Comments:**
  > **username_0**: thanks! So in fact instead of using a point estimator or the fully bayesian approach, you still want to have a distribution (and not point estimates) but instead of applying the true bayesian inference, you formulate the task as an optimization. The intuitive explanation of the free energy term might then be s.th. like: we of course want to increase the likelihood of the parameters given the data but at the same time want to stay close to the priors? Optimizing this term gives us a good approx. of the true posterior  
  > **username_1**: @manu Yes, exactly, I would say that's the typical intuition interpretation of what is going on when you optimize the ELBO.  
  > **username_0**: Already, thanks so much! Just one more question - you said that the ELBO is equivalent to the KL up to some constant. What exactly is the constant? or does the marginal likelihood that becomes by calculating derivative refers to this constant?  
  > **username_1**: The constant is the (log of the) evidence, as you had said in the original version of the post. The ELBO is **equal to** (the **negative** of the) "KL divergence between the posterior and the VD" + "the evidence": that's why **maximizing** the ELBO is equivalent to the original optimization problem of **minimizing** the "KL divergence between the posterior and the VD". As you also had said, the evidence is a constant with respect to the **variational parameters**, i.e. the mean and variance of the VD, which is what you want to find.  
  
  ---
  
  