Answers.Body: <p>In <strong>Bayesian statistics</strong>, as opposed to frequentist statistics, you can model the parameters as random variables. <strong>Bayesian machine learning</strong> is the application of Bayesian statistics in the context of machine learning. The specific application of Bayesian statistics to learning in neural networks is denoted as <strong>Bayesian deep learning</strong>.</p>
  <p>Even just in the simple example of <strong>maximum a posteriori (MAP)</strong> estimation, you model the weights as random variables. For example, you can view <strong>weight decay</strong> in neural networks as MAP estimation when you put a prior on the weights, i.e. you assume that the weights are random variables or samples from a distribution. <a href="https://ai.stackexchange.com/a/16732/2444">Here</a> you can find more details about MAP estimation and how it's different from MLE.</p>
  <p>In MAP, we only find a <strong>point estimate</strong> of the weights. However, we could also learn a <strong>probability distribution over the weights</strong>. If you do this with neural networks, we have <strong>Bayesian neural networks</strong>. Two of the first people that worked on this were <a href="https://authors.library.caltech.edu/13793/1/MACnc92b.pdf" rel="nofollow noreferrer">David J. C. MacKay</a> and <a href="https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.446.9306&amp;rep=rep1&amp;type=pdf" rel="nofollow noreferrer">Radford M. Neal</a>, but only very recently did Bayesian neural networks really become popular, with <a href="https://papers.nips.cc/paper/2011/hash/7eb3c8be3d411e8ebfab08eba5f49632-Abstract.html" rel="nofollow noreferrer">Alex Graves</a> and <a href="http://proceedings.mlr.press/v37/blundell15.pdf" rel="nofollow noreferrer">Blundel et al</a>. Anyway, there are different approaches to Bayesian learning in neural networks. There are techniques based on Monte Carlo methods (e.g. Neal) or variational inference (e.g. Graves and Blundell).</p>
  <p>So, what are the advantages of viewing weights as random variables? Well, you can derive MAP and Bayesian neural networks. Basically, all the advantages of Bayesian statistics.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-23T08:44:35.377Z
Answers.Text: Nice answer, thanks! Do you have any practical examples of where BNNs are used nowadays, maybe even more successful than other alternatives?
Answers.UserDisplayName: Mathy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-23T13:53:16.957Z
Answers.Text: @Mathy Honestly, no. It's been a while since I worked with them, so I forgot a few things. Maybe that could be a good question for our site, so I recommend that you ask it ;) Make sure to you use the tags [tag:state-of-the-art] and [tag:reference-request] and/or to provide some context (e.g. a brief description of BNNs).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-25T08:29:13.977Z
Answers.Text: Thank you for the idea; just done so ;-) https://ai.stackexchange.com/questions/36040/todays-practicality-of-bayesian-neural-networks
Answers.UserDisplayName: Mathy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-16T09:50:24.390Z
Answers.LastActivityDate: 2022-01-16T10:00:31.180Z
Answers.LastEditDate: 2022-01-16T10:00:31.180Z
Answers.UserDisplayName: nbro
Body: <p>In artificial intelligence, especially in machine learning, the <a href="https://ai.stackexchange.com/a/30381/18758">inputs</a> and outputs of neurons in a neural network can be viewed as random variables.</p>
  <p>And this view is highly useful in many ways. The usefulness of this view manifests in applying various mathematical results of random variables in understanding outputs and other aspects of neural networks. Other aspects include loss functions also.</p>
  <p>I want to know whether it is useful to view weights of neural networks as random variables? What does the literature say about this view?</p>
  
Tags: <neural-networks><weights><random-variable>
Title: Is there any advantage in viewing weights of a neural network as random variables?
UserDisplayName: hanugm
ThreadText: # Is there any advantage in viewing weights of a neural network as random variables?
  
  (Asked by: username_0 on 2022-01-16)
  
  In artificial intelligence, especially in machine learning, the [inputs](https://ai.stackexchange.com/a/30381/18758) and outputs of neurons in a neural network can be viewed as random variables.
  
  And this view is highly useful in many ways. The usefulness of this view manifests in applying various mathematical results of random variables in understanding outputs and other aspects of neural networks. Other aspects include loss functions also.
  
  I want to know whether it is useful to view weights of neural networks as random variables? What does the literature say about this view?
  
  **Tags:** neural-networks, weights, random-variable
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-01-16. Score: 2
  
  In **Bayesian statistics**, as opposed to frequentist statistics, you can model the parameters as random variables. **Bayesian machine learning** is the application of Bayesian statistics in the context of machine learning. The specific application of Bayesian statistics to learning in neural networks is denoted as **Bayesian deep learning**.
  
  Even just in the simple example of **maximum a posteriori (MAP)** estimation, you model the weights as random variables. For example, you can view **weight decay** in neural networks as MAP estimation when you put a prior on the weights, i.e. you assume that the weights are random variables or samples from a distribution. [Here](https://ai.stackexchange.com/a/16732/2444) you can find more details about MAP estimation and how it's different from MLE.
  
  In MAP, we only find a **point estimate** of the weights. However, we could also learn a **probability distribution over the weights**. If you do this with neural networks, we have **Bayesian neural networks**. Two of the first people that worked on this were [David J. C. MacKay](https://authors.library.caltech.edu/13793/1/MACnc92b.pdf) and [Radford M. Neal](https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.446.9306&rep=rep1&type=pdf), but only very recently did Bayesian neural networks really become popular, with [Alex Graves](https://papers.nips.cc/paper/2011/hash/7eb3c8be3d411e8ebfab08eba5f49632-Abstract.html) and [Blundel et al](http://proceedings.mlr.press/v37/blundell15.pdf). Anyway, there are different approaches to Bayesian learning in neural networks. There are techniques based on Monte Carlo methods (e.g. Neal) or variational inference (e.g. Graves and Blundell).
  
  So, what are the advantages of viewing weights as random variables? Well, you can derive MAP and Bayesian neural networks. Basically, all the advantages of Bayesian statistics.
  
  > **Comments:**
  > **username_2**: Nice answer, thanks! Do you have any practical examples of where BNNs are used nowadays, maybe even more successful than other alternatives?  
  > **username_1**: @Mathy Honestly, no. It's been a while since I worked with them, so I forgot a few things. Maybe that could be a good question for our site, so I recommend that you ask it ;) Make sure to you use the tags [tag:state-of-the-art] and [tag:reference-request] and/or to provide some context (e.g. a brief description of BNNs).  
  > **username_2**: Thank you for the idea; just done so ;-) https://ai.stackexchange.com/questions/36040/todays-practicality-of-bayesian-neural-networks  
  
  ---
  
  