Answers.Body: <p>You have two dependent variables <span class="math-container">$a$</span> and <span class="math-container">$w$</span>. So, there is a joint distribution <span class="math-container">$p(w, a)$</span>. You can make a marginalization by one of them, pretty much as you did in your second formula.
  <span class="math-container">$$p(w) = \int p(w, a)da$$</span>
  <span class="math-container">$$p(w) = \int p(w | a)p(a)da$$</span></p>
  <p>The only difference in this case, the calculation made for the specific point <span class="math-container">$x_i, y_i$</span>, which is empathized by sub-index on <span class="math-container">$p_i$</span> and conditioning on <span class="math-container">$D_i$</span></p>
  <p>The key thing is that we can calculate the target distribution in many ways. With likelihood, evidence and prior you could indeed find posterior, but they are not always tractable/available. So, that's why in literature we usually differentiate <em>true</em> posterior and approximate posterior (or just posterior). Usually we get <span class="math-container">$p(w|d)$</span> with some form of approximation, but in the paper authors decide to get closed-form solution. That's why it was useful to represent in another way with intermediate distribution <span class="math-container">$a$</span>. This would allow them to get closed-form for different activation functions.</p>
  <p>So it's <em>posterior</em> in the sense of general context, not that specific formula, I think your limits of Bayesian knowledge is just fine</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-23T10:46:23.330Z
Answers.Text: This doesn't seem to answer the main question. Yes, sure, you're explaining how marginalization works, but $p(w)$ typically denotes the marginal probability distribution (e.g. the prior) over $w$ and the posterior is typically denoted by $p(w \mid D)$, while the author of the paper really calls $p(w)$ the "true posterior distribution". How do you explain this? If that's a posterior, you can express it in terms of the likelihood, prior and evidence. I think that's the main question/doubt of the OP is.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-23T10:53:07.293Z
Answers.Text: @m_3464gh Please, don't ask other questions in the comments if you already modified [this post](https://ai.stackexchange.com/q/26950/2444) to ask that question. It's better you just ask the answerer to answer the other question by linking him to it.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-23T10:59:55.693Z
Answers.Text: @nbro, yes thats exactly the main question here.
Answers.UserDisplayName: f_3464gh
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-23T11:00:03.610Z
Answers.Text: Thanks for answering @Kirill Fedyanin - With my knowledge of Bayesian statistics, the true posterior can always be calculated via likelihood prior / evidence. I can't see this in the referred paper. I also wonder why $a$ is used for marginalisation instead of $w$, because we want to get the posterior over the weights - but i already asked this question here if you have any idea on that: https://ai.stackexchange.com/questions/26950/bayesian-perceptron-why-to-marginalize-over-neurons-output-instead-of-its-wei
Answers.UserDisplayName: f_3464gh
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-23T14:33:40.790Z
Answers.Text: @m_3464gh, I updated the answer
Answers.UserDisplayName: Kirill Fedyanin
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-23T10:36:52.817Z
Answers.LastActivityDate: 2021-03-23T14:33:20.490Z
Answers.LastEditDate: 2021-03-23T14:33:20.490Z
Answers.UserDisplayName: Kirill Fedyanin
Answers.Body: <p>Thanks for asking the question. I'm the author of the paper.</p>
  <p>The key point is that the weights <span class="math-container">$w$</span> cannot be updated directly with the new data as <span class="math-container">$w$</span> is not directly related with the output <span class="math-container">$y$</span> (see equation (1)). First, it is necessary to update <span class="math-container">$a$</span> first, which actually is directly related with <span class="math-container">$y$</span> via the activation function, i.e., <span class="math-container">$y = f(a)$</span> with <span class="math-container">$f(.)$</span> being the activation function (see equation (2)). Hence, you will find the Bayes rule in equation (16) with all the quantities you are missing (likelihood, prior, etc.). This equation is used to calculate the posterior <span class="math-container">$p(a|D_i)$</span>. This posterior is then used to update the weights by plugging <span class="math-container">$p(a|D_i)$</span> into equation (14). The solution of equation (14), i.e., the posterior mean and covariance matrix of <span class="math-container">$w$</span>, is provided by means of equation (22). Algorithm 2 provides a summary of all necessary calculations.</p>
  <p>I hope this answers your question. BTW: we have extended this algorithm to a full neural network. The scientific paper describing the whole procedure is currently under review.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-02T18:12:20.517Z
Answers.LastActivityDate: 2021-05-02T18:12:20.517Z
Answers.UserDisplayName: Marco Huber
Body: <p>I found a very interesting paper on the internet that tries to apply Bayesian inference with a gradient-free online-learning approach: [Bayesian Perceptron: <a href="https://arxiv.org/abs/2009.01730" rel="nofollow noreferrer">Bayesian Perceptron: Towards fully Bayesian Neural Networks</a>.</p>
  <p>I would love to understand this work, but, unfortunately, I am reaching my limits with my Bayesian knowledge. Let us assume that we have the weights <span class="math-container">$\mathcal{w}$</span> of our model and observed the data <span class="math-container">$\mathcal{D}$</span>. Using the Bayes rule, we obtain the posterior according to <span class="math-container">$$p(\mathcal{w}|D)=\frac{p(D|\mathcal{w})p(\mathcal{w})}{p(D)}$$</span>.</p>
  <p>In words: we update our prior belief over our weights by multiplying the prior with the likelihood and divide everything by the evidence. In order to calculate the true posterior, we would need to calculate the evidence by marginalizing over (intergrating out) our unknown parameters. This gives the integral <span class="math-container">$$p(D) = \int p(D|\mathbf{w})p(\mathbf{w})dw$$</span>.</p>
  <p>So far so good. Now I refer to the paper mentioned above. Here, the approach is presented exemplarily on a neuron whose weighted sum is called <span class="math-container">$a$</span>, which is then given to the activation function <span class="math-container">$f(.)$</span>. Moreover it is assumed that <span class="math-container">$\mathbf{w}\sim N (\mu_w, \mathbf{C}_w)$</span>. Because of the linearity, it can be exploited that also <span class="math-container">$\mathbf{a}\sim N (\mu_a, \mathbf{C}_a)$</span>.</p>
  <p>What I am confused about now is formula (14), which seems to show the compute the true posterior:
  <span class="math-container">$$p(w) = \int p(a, w|D_i)da = \int p(w|a, D_i)p(a|D_i)da$$</span></p>
  <p>How is this formula of the posterior compatible with the Bayes Theorem? Where is the evidence, likelihood and prior?</p>
  
Tags: <machine-learning><papers><bayesian-neural-networks><bayes-theorem>
Title: Bayesian Perceptron: How is it compatible to Bayes Theorem?
UserDisplayName: f_3464gh
ThreadText: # Bayesian Perceptron: How is it compatible to Bayes Theorem?
  
  (Asked by: username_0 on 2021-03-23)
  
  I found a very interesting paper on the internet that tries to apply Bayesian inference with a gradient-free online-learning approach: [Bayesian Perceptron: [Bayesian Perceptron: Towards fully Bayesian Neural Networks](https://arxiv.org/abs/2009.01730).
  
  I would love to understand this work, but, unfortunately, I am reaching my limits with my Bayesian knowledge. Let us assume that we have the weights $\mathcal{w}$ of our model and observed the data $\mathcal{D}$. Using the Bayes rule, we obtain the posterior according to $$p(\mathcal{w}|D)=\frac{p(D|\mathcal{w})p(\mathcal{w})}{p(D)}$$.
  
  In words: we update our prior belief over our weights by multiplying the prior with the likelihood and divide everything by the evidence. In order to calculate the true posterior, we would need to calculate the evidence by marginalizing over (intergrating out) our unknown parameters. This gives the integral $$p(D) = \int p(D|\mathbf{w})p(\mathbf{w})dw$$.
  
  So far so good. Now I refer to the paper mentioned above. Here, the approach is presented exemplarily on a neuron whose weighted sum is called $a$, which is then given to the activation function $f(.)$. Moreover it is assumed that $\mathbf{w}\sim N (\mu\_w, \mathbf{C}\_w)$. Because of the linearity, it can be exploited that also $\mathbf{a}\sim N (\mu\_a, \mathbf{C}\_a)$.
  
  What I am confused about now is formula (14), which seems to show the compute the true posterior:
  $$p(w) = \int p(a, w|D\_i)da = \int p(w|a, D\_i)p(a|D\_i)da$$
  
  How is this formula of the posterior compatible with the Bayes Theorem? Where is the evidence, likelihood and prior?
  
  **Tags:** machine-learning, papers, bayesian-neural-networks, bayes-theorem
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2021-05-02. Score: 3 | Accepted answer
  
  Thanks for asking the question. I'm the author of the paper.
  
  The key point is that the weights $w$ cannot be updated directly with the new data as $w$ is not directly related with the output $y$ (see equation (1)). First, it is necessary to update $a$ first, which actually is directly related with $y$ via the activation function, i.e., $y = f(a)$ with $f(.)$ being the activation function (see equation (2)). Hence, you will find the Bayes rule in equation (16) with all the quantities you are missing (likelihood, prior, etc.). This equation is used to calculate the posterior $p(a|D\_i)$. This posterior is then used to update the weights by plugging $p(a|D\_i)$ into equation (14). The solution of equation (14), i.e., the posterior mean and covariance matrix of $w$, is provided by means of equation (22). Algorithm 2 provides a summary of all necessary calculations.
  
  I hope this answers your question. BTW: we have extended this algorithm to a full neural network. The scientific paper describing the whole procedure is currently under review.
  
  
  ---
  
  ### Answer by username_1 on 2021-03-23. Score: 0
  
  You have two dependent variables $a$ and $w$. So, there is a joint distribution $p(w, a)$. You can make a marginalization by one of them, pretty much as you did in your second formula.
  $$p(w) = \int p(w, a)da$$
  $$p(w) = \int p(w | a)p(a)da$$
  
  The only difference in this case, the calculation made for the specific point $x\_i, y\_i$, which is empathized by sub-index on $p\_i$ and conditioning on $D\_i$
  
  The key thing is that we can calculate the target distribution in many ways. With likelihood, evidence and prior you could indeed find posterior, but they are not always tractable/available. So, that's why in literature we usually differentiate *true* posterior and approximate posterior (or just posterior). Usually we get $p(w|d)$ with some form of approximation, but in the paper authors decide to get closed-form solution. That's why it was useful to represent in another way with intermediate distribution $a$. This would allow them to get closed-form for different activation functions.
  
  So it's *posterior* in the sense of general context, not that specific formula, I think your limits of Bayesian knowledge is just fine
  
  > **Comments:**
  > **username_2**: This doesn't seem to answer the main question. Yes, sure, you're explaining how marginalization works, but $p(w)$ typically denotes the marginal probability distribution (e.g. the prior) over $w$ and the posterior is typically denoted by $p(w \mid D)$, while the author of the paper really calls $p(w)$ the "true posterior distribution". How do you explain this? If that's a posterior, you can express it in terms of the likelihood, prior and evidence. I think that's the main question/doubt of the OP is.  
  > **username_2**: @m_3464gh Please, don't ask other questions in the comments if you already modified [this post](https://ai.stackexchange.com/q/26950/2444) to ask that question. It's better you just ask the answerer to answer the other question by linking him to it.  
  > **username_0**: @username_2, yes thats exactly the main question here.  
  > **username_0**: Thanks for answering @username_1 - With my knowledge of Bayesian statistics, the true posterior can always be calculated via likelihood prior / evidence. I can't see this in the referred paper. I also wonder why $a$ is used for marginalisation instead of $w$, because we want to get the posterior over the weights - but i already asked this question here if you have any idea on that: https://ai.stackexchange.com/questions/26950/bayesian-perceptron-why-to-marginalize-over-neurons-output-instead-of-its-wei  
  > **username_1**: @m_3464gh, I updated the answer  
  
  ---
  
  