Answers.Body: <blockquote>
  <p>We want a distribution over <span class="math-container">$w$</span>, don't we?</p>
  </blockquote>
  <p>Yes. You want to obtain a distribution over the parameters, which models the uncertainty about the parameters. This distribution over the parameters can <em>induce</em> a probability distribution over the possible functions consistent with your data.</p>
  <blockquote>
  <p>Why is <span class="math-container">$a$</span> integrated out here and not <span class="math-container">$w$</span>?</p>
  </blockquote>
  <p>This is just the definition of <a href="https://en.wikipedia.org/wiki/Marginal_distribution" rel="nofollow noreferrer"><strong>marginalization</strong></a>.</p>
  <p>For simplicity, consider 2 discrete random variables <span class="math-container">$X$</span> and <span class="math-container">$Y$</span>, which can take values <span class="math-container">$x_i$</span> and <span class="math-container">$y_i$</span>, respectively, and for which you can have (at least conceptually) a <a href="https://en.wikipedia.org/wiki/Joint_probability_distribution" rel="nofollow noreferrer">joint distribution</a> <span class="math-container">$p_{(X, Y)}(x, y)$</span>. Then the marginal <em>probability</em> that <span class="math-container">$X$</span> takes the value <span class="math-container">$x_i$</span> <strong>independently</strong> of any outcome of the r.v. <span class="math-container">$Y$</span> (and this is the intuition behind the marginal distribution!) is given by</p>
  <p><span class="math-container">$${\displaystyle p_{X}(x_{i})=\sum _{j}p(x_{i},y_{j})}, \tag{1}\label{1}$$</span></p>
  <p>So, in the case of equation 14 of the paper, there's the assumption that you have <span class="math-container">$p(w, a)$</span>. Then <span class="math-container">$p(w)$</span> is a marginal distribution over <span class="math-container">$w$</span> (the parameters or weights of the neural network), independently of the values of the activations. Of course, <span class="math-container">$w$</span> and <span class="math-container">$a$</span> can be dependent or correlated, and that's why you also have <span class="math-container">$p(w, a)$</span>, i.e. the probability distribution that <em>describes</em> how these two r.v.s <em>vary together</em>, but what you want to know is the probability distribution over <span class="math-container">$w$</span> independently of <span class="math-container">$a$</span>, so you &quot;marginalize out&quot; the possible values of <span class="math-container">$a$</span> (not <span class="math-container">$w$</span>).</p>
  <p>Remember that <span class="math-container">$p(x, y) = p(x \mid y) p(y) = p(y \mid x) p(x)$</span> (this is known as the <a href="https://en.wikipedia.org/wiki/Chain_rule_(probability)" rel="nofollow noreferrer"><strong>chain rule</strong></a> of probability for two r.v.s, which leads to the Bayes' theorem), so you could write equation (\ref{1}) as follows</p>
  <p><span class="math-container">$${\displaystyle p_{X}(x_{i})=\sum _{j}p(x_{i},y_{j})} = \sum _{j}p(x_{i} \mid y_{j}) p(y_j), \tag{2}\label{2}$$</span></p>
  <p>In the case of real-valued r.v.s, you will have an integral rather than a summation, but the idea is roughly the same.</p>
  <blockquote>
  <p>But without marginalization over <span class="math-container">$w$</span> there is still uncertainty inside <span class="math-container">$w$</span>.</p>
  </blockquote>
  <p>What we want to know is <span class="math-container">$p(w)$</span>, i.e. a distribution over <span class="math-container">$w$</span>, which models the <strong>uncertainty</strong> about the possible values that <span class="math-container">$w$</span> can take. So, once you know <span class="math-container">$p(w)$</span>, depending on its shape, we will have more or less uncertainty about the possible values of <span class="math-container">$w$</span>. For example, let's say <span class="math-container">$p(w)$</span> is a Gaussian distribution. Then, depending on the variance, you will have more or less uncertainty about the possible values of <span class="math-container">$w$</span>.</p>
  <p>Your doubt seems to be that you thought that marginalization is a way to not have uncertainty, but that's not true. So, the goal in <em>Bayesian machine/deep learning</em> is not to be certain about the values of the parameters, but to model the uncertainty about the values of the parameters.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-23T11:23:39.513Z
Answers.Text: By the way, I didn't read that paper (maybe later if I have more time I will read it). This answer is based on my knowledge of probability theory. So, I am not really sure why the author is doing what is doing, but, conceptually, it's possible (I think). I would need to read the paper to fully answer the question in the title (which is slightly different than, although related to, the questions in the body).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-23T11:08:26.367Z
Answers.LastActivityDate: 2021-03-23T15:53:47.767Z
Answers.LastEditDate: 2021-03-23T15:53:47.767Z
Answers.UserDisplayName: nbro
Body: <p>I found a very interesting paper on the internet that tries to apply Bayesian inference with a gradient-free online-learning approach: <a href="https://arxiv.org/abs/2009.01730" rel="nofollow noreferrer">Bayesian Perceptron: Towards fully Bayesian Neural Networks</a>.</p>
  <p>I would love to understand this work, but unfortunately I am reaching my limits with my Bayesian knowledge. Let us assume that we have the weights <span class="math-container">$\mathcal{w}$</span> of our model and observed the data <span class="math-container">$\mathcal{D}$</span>. Using the Bayes rule, we obtain the posterior according to <span class="math-container">$$p(\mathcal{w}|D)=\frac{p(D|\mathcal{w})p(\mathcal{w})}{p(D)}$$</span>.</p>
  <p>In words: we update our prior belief over our weights by multiplying the prior with the likelihood and divide everything by the evidence. In order to calculate the true posterior, we would need to calculate the evidence by marginalizing over (intergrating out) our unknown parameters. This gives the integral <span class="math-container">$$p(D) = \int p(D|\mathbf{w})p(\mathbf{w})dw$$</span>.</p>
  <p>So far so good. Now I refer to the paper mentioned above. Here, the approach is presented exemplarily on a neuron whose weighted sum is called <span class="math-container">$a$</span>, which is then given to the activation function <span class="math-container">$f(.)$</span>. Moreover it is assumed that <span class="math-container">$\mathbf{w}\sim N (\mu_w, \mathbf{C}_w)$</span>. Because of the linearity it can be exploited that also <span class="math-container">$\mathbf{a}\sim N (\mu_a, \mathbf{C}_a)$</span>.</p>
  <p>What I am confused about now is formula (14), which seems to show the compute the true posterior:
  <span class="math-container">$$p(w) = \int p(a, w|D_i)da = \int p(w|a, D_i)p(a|D_i)da$$</span></p>
  <p>Why is <span class="math-container">$a$</span> integrated out here and not <span class="math-container">$w$</span>? We want a distribution over <span class="math-container">$w$</span>, don't we? But without marginalization over <span class="math-container">$w$</span> there is still uncertainty inside <span class="math-container">$w$</span></p>
  <p>Glad about any help and food for thought;)</p>
  
Tags: <machine-learning><papers><bayesian-neural-networks><bayes-theorem><probability-theory>
Title: Bayesian Perceptron: Why to marginalize over neuron's output instead of it's weights?
UserDisplayName: f_3464gh
ThreadText: # Bayesian Perceptron: Why to marginalize over neuron's output instead of it's weights?
  
  (Asked by: username_0 on 2021-03-22)
  
  I found a very interesting paper on the internet that tries to apply Bayesian inference with a gradient-free online-learning approach: [Bayesian Perceptron: Towards fully Bayesian Neural Networks](https://arxiv.org/abs/2009.01730).
  
  I would love to understand this work, but unfortunately I am reaching my limits with my Bayesian knowledge. Let us assume that we have the weights $\mathcal{w}$ of our model and observed the data $\mathcal{D}$. Using the Bayes rule, we obtain the posterior according to $$p(\mathcal{w}|D)=\frac{p(D|\mathcal{w})p(\mathcal{w})}{p(D)}$$.
  
  In words: we update our prior belief over our weights by multiplying the prior with the likelihood and divide everything by the evidence. In order to calculate the true posterior, we would need to calculate the evidence by marginalizing over (intergrating out) our unknown parameters. This gives the integral $$p(D) = \int p(D|\mathbf{w})p(\mathbf{w})dw$$.
  
  So far so good. Now I refer to the paper mentioned above. Here, the approach is presented exemplarily on a neuron whose weighted sum is called $a$, which is then given to the activation function $f(.)$. Moreover it is assumed that $\mathbf{w}\sim N (\mu\_w, \mathbf{C}\_w)$. Because of the linearity it can be exploited that also $\mathbf{a}\sim N (\mu\_a, \mathbf{C}\_a)$.
  
  What I am confused about now is formula (14), which seems to show the compute the true posterior:
  $$p(w) = \int p(a, w|D\_i)da = \int p(w|a, D\_i)p(a|D\_i)da$$
  
  Why is $a$ integrated out here and not $w$? We want a distribution over $w$, don't we? But without marginalization over $w$ there is still uncertainty inside $w$
  
  Glad about any help and food for thought;)
  
  **Tags:** machine-learning, papers, bayesian-neural-networks, bayes-theorem, probability-theory
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-03-23. Score: 1
  
  > We want a distribution over $w$, don't we?
  
  Yes. You want to obtain a distribution over the parameters, which models the uncertainty about the parameters. This distribution over the parameters can *induce* a probability distribution over the possible functions consistent with your data.
  
  > Why is $a$ integrated out here and not $w$?
  
  This is just the definition of [**marginalization**](https://en.wikipedia.org/wiki/Marginal_distribution).
  
  For simplicity, consider 2 discrete random variables $X$ and $Y$, which can take values $x\_i$ and $y\_i$, respectively, and for which you can have (at least conceptually) a [joint distribution](https://en.wikipedia.org/wiki/Joint_probability_distribution) $p\_{(X, Y)}(x, y)$. Then the marginal *probability* that $X$ takes the value $x\_i$ **independently** of any outcome of the r.v. $Y$ (and this is the intuition behind the marginal distribution!) is given by
  
  $${\displaystyle p\_{X}(x\_{i})=\sum \_{j}p(x\_{i},y\_{j})}, \tag{1}\label{1}$$
  
  So, in the case of equation 14 of the paper, there's the assumption that you have $p(w, a)$. Then $p(w)$ is a marginal distribution over $w$ (the parameters or weights of the neural network), independently of the values of the activations. Of course, $w$ and $a$ can be dependent or correlated, and that's why you also have $p(w, a)$, i.e. the probability distribution that *describes* how these two r.v.s *vary together*, but what you want to know is the probability distribution over $w$ independently of $a$, so you "marginalize out" the possible values of $a$ (not $w$).
  
  Remember that $p(x, y) = p(x \mid y) p(y) = p(y \mid x) p(x)$ (this is known as the [**chain rule**](https://en.wikipedia.org/wiki/Chain_rule_(probability)) of probability for two r.v.s, which leads to the Bayes' theorem), so you could write equation (\ref{1}) as follows
  
  $${\displaystyle p\_{X}(x\_{i})=\sum \_{j}p(x\_{i},y\_{j})} = \sum \_{j}p(x\_{i} \mid y\_{j}) p(y\_j), \tag{2}\label{2}$$
  
  In the case of real-valued r.v.s, you will have an integral rather than a summation, but the idea is roughly the same.
  
  > But without marginalization over $w$ there is still uncertainty inside $w$.
  
  What we want to know is $p(w)$, i.e. a distribution over $w$, which models the **uncertainty** about the possible values that $w$ can take. So, once you know $p(w)$, depending on its shape, we will have more or less uncertainty about the possible values of $w$. For example, let's say $p(w)$ is a Gaussian distribution. Then, depending on the variance, you will have more or less uncertainty about the possible values of $w$.
  
  Your doubt seems to be that you thought that marginalization is a way to not have uncertainty, but that's not true. So, the goal in *Bayesian machine/deep learning* is not to be certain about the values of the parameters, but to model the uncertainty about the values of the parameters.
  
  > **Comments:**
  > **username_1**: By the way, I didn't read that paper (maybe later if I have more time I will read it). This answer is based on my knowledge of probability theory. So, I am not really sure why the author is doing what is doing, but, conceptually, it's possible (I think). I would need to read the paper to fully answer the question in the title (which is slightly different than, although related to, the questions in the body).  
  
  ---
  
  