Answers.Body: <p>Yes, there is some research on this topic. It's often called Bayesian machine learning or Bayesian deep learning (but I don't think this is a good name because there are models that aren't really based on a direct application of Bayesian statistics). Some ML/DL models that provide some kind of uncertainty estimation are, for example, <a href="https://arxiv.org/abs/1506.02142" rel="nofollow noreferrer">Monte Carlo Dropout (MC dropout)</a> or <a href="https://arxiv.org/pdf/1505.05424.pdf" rel="nofollow noreferrer">Bayesian neural networks</a>. In theory, these techniques look promising. In practice, I don't think they are the ultimate solution to the problem of uncertainty estimation in deep learning. In fact, e.g. in the case of Bayesian neural networks, they have some disadvantages, such as more parameters to tweak and save.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-23T08:59:24.217Z
Answers.Text: Do you know of any practical examples where the disadvantages you mention really get clear?
Answers.UserDisplayName: Mathy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-23T13:47:58.280Z
Answers.Text: @Mathy Off the top of my head, no. I only worked with relatively small BNNs (1 million parameters? I don't remember, honestly) in the past (it was already several months ago when I worked with them - I don't know the SOTA now - But probably nothing changed much). But let's say that in the future models are of size like GPT-3's size, then this could be a problem, as you may need to double the number of parameters, so I assume that this will cause issues, in addition to the very likely longer training times to fit all those parameters.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-06T12:14:19.057Z
Answers.LastActivityDate: 2020-05-06T12:14:19.057Z
Answers.UserDisplayName: nbro
Body: <p>Is there any research on machine learning models that provide uncertainty estimation?</p>
  
  <p>If I train a denoising autoencoder on words and put through a noised word, I'd like it to return a certainty that it is correct given the distribution of data it has been trained on. </p>
  
  <p>Answering these questions or metrics for uncertainty are both things I am curious about. Just general ways for models to just say "I'm not sure" when it receives something far outside the inputs it's been trained to approximate.</p>
  
Tags: <machine-learning><bayesian-deep-learning><bayesian-neural-networks><uncertainty-quantification>
Title: Is there any research on models that provide uncertainty estimation?
UserDisplayName: user8714896
ThreadText: # Is there any research on models that provide uncertainty estimation?
  
  (Asked by: username_0 on 2020-05-06)
  
  Is there any research on machine learning models that provide uncertainty estimation?
  
  If I train a denoising autoencoder on words and put through a noised word, I'd like it to return a certainty that it is correct given the distribution of data it has been trained on.
  
  Answering these questions or metrics for uncertainty are both things I am curious about. Just general ways for models to just say "I'm not sure" when it receives something far outside the inputs it's been trained to approximate.
  
  **Tags:** machine-learning, bayesian-deep-learning, bayesian-neural-networks, uncertainty-quantification
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-05-06. Score: 2 | Accepted answer
  
  Yes, there is some research on this topic. It's often called Bayesian machine learning or Bayesian deep learning (but I don't think this is a good name because there are models that aren't really based on a direct application of Bayesian statistics). Some ML/DL models that provide some kind of uncertainty estimation are, for example, [Monte Carlo Dropout (MC dropout)](https://arxiv.org/abs/1506.02142) or [Bayesian neural networks](https://arxiv.org/pdf/1505.05424.pdf). In theory, these techniques look promising. In practice, I don't think they are the ultimate solution to the problem of uncertainty estimation in deep learning. In fact, e.g. in the case of Bayesian neural networks, they have some disadvantages, such as more parameters to tweak and save.
  
  > **Comments:**
  > **username_2**: Do you know of any practical examples where the disadvantages you mention really get clear?  
  > **username_1**: @Mathy Off the top of my head, no. I only worked with relatively small BNNs (1 million parameters? I don't remember, honestly) in the past (it was already several months ago when I worked with them - I don't know the SOTA now - But probably nothing changed much). But let's say that in the future models are of size like GPT-3's size, then this could be a problem, as you may need to double the number of parameters, so I assume that this will cause issues, in addition to the very likely longer training times to fit all those parameters.  
  
  ---
  
  