Answers.Body: <p>The <a href="https://en.wikipedia.org/wiki/Likelihood_function" rel="nofollow noreferrer">likelihood</a> depends on the task that you are solving, so this is similar to traditional neural networks (in fact, even these neural networks have a probabilistic/Bayesian interpretation!).</p>
  <ul>
  <li><p>For binary classification, you should probably use a Bernoulli, which, in practice, corresponds to using a sigmoid with a binary cross-entropy (you can show that the minimization of the cross-entropy is equivalent to the maximization of Bernoulli p.m.f.)</p>
  </li>
  <li><p>If it's a multi-class classification problem, you should use a categorical distribution, which corresponds to a softmax with a categorical cross-entropy; see e.g. <a href="https://github.com/tensorflow/probability/blob/master/tensorflow_probability/examples/bayesian_neural_network.py" rel="nofollow noreferrer">this implementation</a></p>
  </li>
  <li><p>If it's a regression problem, the likelihood could be a Gaussian (which is equivalent to using the MSE: <a href="https://www.youtube.com/watch?v=vEPQNwxd1Y4" rel="nofollow noreferrer">you can also show this</a>).</p>
  </li>
  </ul>
  <p>In the case of <a href="http://proceedings.mlr.press/v37/blundell15.pdf" rel="nofollow noreferrer">(mean-field) <strong>variational</strong> BNNs (VBNNs)</a> (note that not all neural networks denoted as BNNs are VBNNs, but here I will only focus on VBNNs), rather than directly performing Bayesian inference (i.e. applying the Bayes theorem directly), you will perform approximate inference and, more specifically, you cast the inference problem as an equivalent optimization problem, where you typically optimize the loss function known as the <strong>evidence lower bound (ELBO)</strong>, which is composed of two terms</p>
  <ol>
  <li>the (log-)likelihood of the parameters (so the Bernoulli/categorical/Gaussian)</li>
  <li>the KL divergence (the regularization part)</li>
  </ol>
  <p>Theoretically, given that I am not a statistician, I cannot tell you right now why we can have a Bernoulli likelihood and a Gaussian prior (which is typically the case and, more specifically, in the case of <a href="http://proceedings.mlr.press/v37/blundell15.pdf" rel="nofollow noreferrer">this paper</a>, they assume the weights to be independent, which is kind of a strong assumption but simplifies the computations, so this leads to a factorized multi-variate Gaussian over the weights), but this is what I had seen in practice.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-20T10:07:36.360Z
Answers.Text: Does this mean that if I have a bayesian neural network with sigmoid with a binary cross-entropy (which in fact is similar to bernoulli) we could use a prior that is conjugate to the likelihood (i.e. bernoulli) and thus avoid to calculate the evidence? If I understood correctly, your examples are comparisons, aren't they? So I don't have to explicitly define a Bernoulli distribution, but the use of sigmoid with a binary cross-entropy is comparable to an experiment where we assume the data is Bernoulli distributed?
Answers.UserDisplayName: Micha Christ
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-20T23:12:15.947Z
Answers.Text: @MichaChrist I don't know the answer to your first question, but, typically, in BNNs we don't calculate the evidence anyway because it's intractable. To answer your second question, I think the answer is "yes", but I am not fully about the accurateness of that statement.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-17T10:33:04.847Z
Answers.LastActivityDate: 2021-03-17T12:15:47.167Z
Answers.LastEditDate: 2021-03-17T12:15:47.167Z
Answers.UserDisplayName: nbro
Body: <p>I'm trying to understand the concept behind BNN.
  Their are based on the Bayes Theorem:</p>
  <p><span class="math-container">$$p(w \mid \text{data}) = \frac{p(\text{data} \mid w)*p(w)}{p(\text{data})}$$</span></p>
  <p>which boils down to</p>
  <p><span class="math-container">$$\text{posterior} = \frac{\text{likelihood} * \text{prior}}{\text{evidence}}.$$</span></p>
  <p>I understand that if we assume a Gaussian distribution for the model, the likelihood function comprises a product (or sum if we use log) of each data point inserted into the Gaussian pdf. The parameters which we can change are <span class="math-container">$\mu$</span> and <span class="math-container">$\sigma^2$</span>. We want to increase the likelihood because higher values of the Gaussian pdf means higher probability.</p>
  <p>How do things work when using a neural network? I assume that the likelihood function looks s.th. like inserting each data point inside the calculation (weighted sums) of the neural network. But does the neural network need a softmax layer at the end so that we can interpret the outputs as probabilities/likelihoods? Or do we measure likelihood by applying some error measurement like cross-entropy or squared-loss?</p>
  
Tags: <bayesian-deep-learning><bayesian-neural-networks><bayes-theorem>
Title: What's the likelihood in Bayesian Neural Networks?
UserDisplayName: Micha Christ
ThreadText: # What's the likelihood in Bayesian Neural Networks?
  
  (Asked by: username_0 on 2021-03-17)
  
  I'm trying to understand the concept behind BNN.
  Their are based on the Bayes Theorem:
  
  $$p(w \mid \text{data}) = \frac{p(\text{data} \mid w)\*p(w)}{p(\text{data})}$$
  
  which boils down to
  
  $$\text{posterior} = \frac{\text{likelihood} \* \text{prior}}{\text{evidence}}.$$
  
  I understand that if we assume a Gaussian distribution for the model, the likelihood function comprises a product (or sum if we use log) of each data point inserted into the Gaussian pdf. The parameters which we can change are $\mu$ and $\sigma^2$. We want to increase the likelihood because higher values of the Gaussian pdf means higher probability.
  
  How do things work when using a neural network? I assume that the likelihood function looks s.th. like inserting each data point inside the calculation (weighted sums) of the neural network. But does the neural network need a softmax layer at the end so that we can interpret the outputs as probabilities/likelihoods? Or do we measure likelihood by applying some error measurement like cross-entropy or squared-loss?
  
  **Tags:** bayesian-deep-learning, bayesian-neural-networks, bayes-theorem
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-03-17. Score: 1
  
  The [likelihood](https://en.wikipedia.org/wiki/Likelihood_function) depends on the task that you are solving, so this is similar to traditional neural networks (in fact, even these neural networks have a probabilistic/Bayesian interpretation!).
  
  * For binary classification, you should probably use a Bernoulli, which, in practice, corresponds to using a sigmoid with a binary cross-entropy (you can show that the minimization of the cross-entropy is equivalent to the maximization of Bernoulli p.m.f.)
  * If it's a multi-class classification problem, you should use a categorical distribution, which corresponds to a softmax with a categorical cross-entropy; see e.g. [this implementation](https://github.com/tensorflow/probability/blob/master/tensorflow_probability/examples/bayesian_neural_network.py)
  * If it's a regression problem, the likelihood could be a Gaussian (which is equivalent to using the MSE: [you can also show this](https://www.youtube.com/watch?v=vEPQNwxd1Y4)).
  
  In the case of [(mean-field) **variational** BNNs (VBNNs)](http://proceedings.mlr.press/v37/blundell15.pdf) (note that not all neural networks denoted as BNNs are VBNNs, but here I will only focus on VBNNs), rather than directly performing Bayesian inference (i.e. applying the Bayes theorem directly), you will perform approximate inference and, more specifically, you cast the inference problem as an equivalent optimization problem, where you typically optimize the loss function known as the **evidence lower bound (ELBO)**, which is composed of two terms
  
  1. the (log-)likelihood of the parameters (so the Bernoulli/categorical/Gaussian)
  2. the KL divergence (the regularization part)
  
  Theoretically, given that I am not a statistician, I cannot tell you right now why we can have a Bernoulli likelihood and a Gaussian prior (which is typically the case and, more specifically, in the case of [this paper](http://proceedings.mlr.press/v37/blundell15.pdf), they assume the weights to be independent, which is kind of a strong assumption but simplifies the computations, so this leads to a factorized multi-variate Gaussian over the weights), but this is what I had seen in practice.
  
  > **Comments:**
  > **username_0**: Does this mean that if I have a bayesian neural network with sigmoid with a binary cross-entropy (which in fact is similar to bernoulli) we could use a prior that is conjugate to the likelihood (i.e. bernoulli) and thus avoid to calculate the evidence? If I understood correctly, your examples are comparisons, aren't they? So I don't have to explicitly define a Bernoulli distribution, but the use of sigmoid with a binary cross-entropy is comparable to an experiment where we assume the data is Bernoulli distributed?  
  > **username_1**: @username_0 I don't know the answer to your first question, but, typically, in BNNs we don't calculate the evidence anyway because it's intractable. To answer your second question, I think the answer is "yes", but I am not fully about the accurateness of that statement.  
  
  ---
  
  