Answers.Body: <p>The main reason should be that Bayesian algorithms naturally incorporate a form of <strong>regularisation</strong> (the <strong>prior</strong>), so they should be <strong>less prone to over-fitting</strong> the small dataset. Of course, the choice of the prior can affect your estimates.</p>
  <p>You can view certain training regimes in machine learning and deep learning as an application of Bayesian statistics: for example, <a href="https://ai.stackexchange.com/a/17671/2444">if you train a neural network with weight decay, this is equivalent to a MAP estimate</a>.</p>
  <p>I guess it might also be computationally easier to perform Bayesian inference with a smaller dataset (although I have never tried to show that this is true: my practical experience in this context is limited to <a href="http://proceedings.mlr.press/v37/blundell15.pdf" rel="nofollow noreferrer">Bayesian neural networks</a>). Of course, the more data you have, in theory, the closer your estimates should be to the actual true values.</p>
  <p><a href="https://math.stackexchange.com/q/2589224/168764">Here</a> you have a similar question.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-13T10:21:50.833Z
Answers.LastActivityDate: 2021-12-13T10:28:51.463Z
Answers.LastEditDate: 2021-12-13T10:28:51.463Z
Answers.UserDisplayName: nbro
Body: <p>I read very often that <a href="https://math.stackexchange.com/q/2589224/168764">Bayesian algorithms work well on small datasets</a>. Why is that? I think it is because they might generalize more, but why is that? </p>
  
  <p>See also <a href="https://lib.dr.iastate.edu/cgi/viewcontent.cgi?article=7636&amp;context=etd" rel="nofollow noreferrer">Investigating the use of Bayesian networks for small dataset problems</a>.</p>
  
Tags: <machine-learning><bayesian-networks><naive-bayes><bayesian-statistics>
Title: Why do Bayesian algorithms work well with small datasets?
UserDisplayName: jennifer ruurs
ThreadText: # Why do Bayesian algorithms work well with small datasets?
  
  (Asked by: username_0 on 2019-11-23)
  
  I read very often that [Bayesian algorithms work well on small datasets](https://math.stackexchange.com/q/2589224/168764). Why is that? I think it is because they might generalize more, but why is that?
  
  See also [Investigating the use of Bayesian networks for small dataset problems](https://lib.dr.iastate.edu/cgi/viewcontent.cgi?article=7636&context=etd).
  
  **Tags:** machine-learning, bayesian-networks, naive-bayes, bayesian-statistics
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-12-13. Score: 1
  
  The main reason should be that Bayesian algorithms naturally incorporate a form of **regularisation** (the **prior**), so they should be **less prone to over-fitting** the small dataset. Of course, the choice of the prior can affect your estimates.
  
  You can view certain training regimes in machine learning and deep learning as an application of Bayesian statistics: for example, [if you train a neural network with weight decay, this is equivalent to a MAP estimate](https://ai.stackexchange.com/a/17671/2444).
  
  I guess it might also be computationally easier to perform Bayesian inference with a smaller dataset (although I have never tried to show that this is true: my practical experience in this context is limited to [Bayesian neural networks](http://proceedings.mlr.press/v37/blundell15.pdf)). Of course, the more data you have, in theory, the closer your estimates should be to the actual true values.
  
  [Here](https://math.stackexchange.com/q/2589224/168764) you have a similar question.
  
  
  ---
  
  