Answers.Body: <p>Off the top of my head, I don't know the very specific definition of <em>confidence interval</em> (or whether it's only defined for the parameters of a model), as I am not a statistician. In any case, intuitively, a confidence interval is an interval (or range) of values where some true value of something (e.g. your parameter) lies. (Confidence intervals are also very related to <em>hypothesis testing</em>, but I will not dwell on this topic here). You can find the specific definition of a confidence interval in any statistics book (e.g. <a href="https://fac.ksu.edu.sa/sites/default/files/677_fr37hij.pdf#page=310" rel="nofollow noreferrer">this one</a>).</p>
  <p>Having said that, I interpret that statement as saying that most machine learning approaches do not take into account any type of <strong>uncertainty</strong> about either the true value of the parameters (e.g. of the neural networks) or the predictions or do not deal with hypothesis testing (recall above that I said that hypothesis testing is closely related to confidence intervals). Typically, in machine learning, you will find a lot of approaches that just provide you with a <a href="https://en.wikipedia.org/wiki/Point_estimation" rel="nofollow noreferrer"><strong>point estimate</strong></a> for the parameters (i.e. you estimate a single number for each parameter). Consequently, your final neural network (or model) just represents a single function, but what if that function is not really correct (which is probably the case given the typically limited amount of data)? In this case, you cannot say anything about your uncertainty of the true target function that you were trying to approximate or about the prediction for a new input.</p>
  <p>Here's where <strong>probabilistic/Bayesian machine learning</strong> (PML) comes into play. Probabilistic machine learning is a relatively new subfield of machine learning that deals with <a href="https://en.wikipedia.org/wiki/Uncertainty_quantification" rel="nofollow noreferrer"><strong>uncertainty quantification/estimation</strong></a> or that uses tools from Bayesian statistics, like the Bayes theorem. If (deep) neural networks are involved, it is also called <strong>Bayesian deep learning</strong> (BDL).</p>
  <p>For a gentle overview of PML, you can read <a href="https://www.nature.com/articles/nature14541" rel="nofollow noreferrer">this paper</a>. If you want to know more about Bayesian neural networks (i.e. neural networks that learn a probability distribution over functions that are consistent with the data), you can read <a href="http://proceedings.mlr.press/v37/blundell15.pdf" rel="nofollow noreferrer">this paper</a>.</p>
  <p>So, nowadays, I wouldn't say that that statement is &quot;true&quot;, although it was probably true when that book was published.  More and more, people in the machine learning community do research in PML and, in particular, BDL, because, if you want to adopt neural networks in areas like healthcare, you need to provide the doctors with some kind of uncertainty quantification of the predictions. Let's say that that a doctor needs to take an action, such as giving some kind of medicine to a patient based on the condition of the patient (e.g. temperature). The doctor doesn't just want to know &quot;yes, give the medicine to the patient&quot;. The doctor wants to have an idea of how confident or uncertain the model is about its prediction. This is also where another subfield of AI comes into play, i.e. <a href="https://ai.stackexchange.com/q/14224/2444"><em>explainable AI</em></a>, but I will not dwell more on this topic here.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-25T13:40:49.877Z
Answers.LastActivityDate: 2021-12-13T09:24:38.057Z
Answers.LastEditDate: 2021-12-13T09:24:38.057Z
Answers.UserDisplayName: nbro
Answers.Body: <p>The confidence interval is a way of quantifying expected error in predictions.</p>
  <p>For example, lets say you are trying to model your dart throwing scoring metrics. If you play many games, and assume the score obtained follows a normal distribution, you can do a statistical calculation known as maximum likelihood estimation which provides you with parameters of the normal distribution that best fits the data. This allows you to quantify the error more precisely, so you can say things like: &quot;95% of my games will be at least 200 points&quot;, or &quot;having a score this good/bad is a 1 in 1000 chance.&quot;</p>
  <p>Much of machine learning is about outputting best predictions given some input, and not about quantifying error. So in the above example, we might just output a single value that best captures the underlying statistic, which in the above case, would be the mean of all scores, which is confirmed by the maximum likelihood estimate approach in statistics.</p>
  <p>A quick online search returns the definition: <a href="https://www.investopedia.com/terms/c/confidenceinterval.asp" rel="nofollow noreferrer">&quot;the probability that a population parameter will fall between a set of values for a certain proportion of times.&quot;</a>, which seems to agree with the above.</p>
  <p>Bar graphs with confidence intervals would be a very clear visual intuition of the idea.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-25T06:09:22.610Z
Answers.LastActivityDate: 2021-08-25T14:45:04.663Z
Answers.LastEditDate: 2021-08-25T14:45:04.663Z
Answers.UserDisplayName: k.c. sayz 'k.c sayz'
Body: <p>Consider the following excerpt from <a href="https://www.deeplearningbook.org/contents/ml.html" rel="nofollow noreferrer">Chapter 5: Machine Learning Basics</a> from the book titled <a href="https://www.deeplearningbook.org/" rel="nofollow noreferrer">Deep Learning</a> (by Aaron Courville et al.)</p>
  <blockquote>
  <p>Machine learning is essentially a form of applied statistics with increased emphasis on the use of computers to statistically estimate complicated functions and a <strong>decreased emphasis on proving conﬁdence intervals around these functions</strong>;</p>
  </blockquote>
  <p>This excerpt says that machine learning focuses on estimating complicated functions, but not on proving conﬁdence intervals around those functions. What is meant by or definition for a confidence interval around a complicated function mentioned here?</p>
  
Tags: <machine-learning><definitions><statistics><probabilistic-machine-learning>
Title: What is the definition of "confidence interval" around a (complicated) function?
UserDisplayName: hanugm
ThreadText: # What is the definition of "confidence interval" around a (complicated) function?
  
  (Asked by: username_0 on 2021-08-25)
  
  Consider the following excerpt from [Chapter 5: Machine Learning Basics](https://www.deeplearningbook.org/contents/ml.html) from the book titled [Deep Learning](https://www.deeplearningbook.org/) (by Aaron Courville et al.)
  
  > Machine learning is essentially a form of applied statistics with increased emphasis on the use of computers to statistically estimate complicated functions and a **decreased emphasis on proving conﬁdence intervals around these functions**;
  
  This excerpt says that machine learning focuses on estimating complicated functions, but not on proving conﬁdence intervals around those functions. What is meant by or definition for a confidence interval around a complicated function mentioned here?
  
  **Tags:** machine-learning, definitions, statistics, probabilistic-machine-learning
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2021-08-25. Score: 1
  
  Off the top of my head, I don't know the very specific definition of *confidence interval* (or whether it's only defined for the parameters of a model), as I am not a statistician. In any case, intuitively, a confidence interval is an interval (or range) of values where some true value of something (e.g. your parameter) lies. (Confidence intervals are also very related to *hypothesis testing*, but I will not dwell on this topic here). You can find the specific definition of a confidence interval in any statistics book (e.g. [this one](https://fac.ksu.edu.sa/sites/default/files/677_fr37hij.pdf#page=310)).
  
  Having said that, I interpret that statement as saying that most machine learning approaches do not take into account any type of **uncertainty** about either the true value of the parameters (e.g. of the neural networks) or the predictions or do not deal with hypothesis testing (recall above that I said that hypothesis testing is closely related to confidence intervals). Typically, in machine learning, you will find a lot of approaches that just provide you with a [**point estimate**](https://en.wikipedia.org/wiki/Point_estimation) for the parameters (i.e. you estimate a single number for each parameter). Consequently, your final neural network (or model) just represents a single function, but what if that function is not really correct (which is probably the case given the typically limited amount of data)? In this case, you cannot say anything about your uncertainty of the true target function that you were trying to approximate or about the prediction for a new input.
  
  Here's where **probabilistic/Bayesian machine learning** (PML) comes into play. Probabilistic machine learning is a relatively new subfield of machine learning that deals with [**uncertainty quantification/estimation**](https://en.wikipedia.org/wiki/Uncertainty_quantification) or that uses tools from Bayesian statistics, like the Bayes theorem. If (deep) neural networks are involved, it is also called **Bayesian deep learning** (BDL).
  
  For a gentle overview of PML, you can read [this paper](https://www.nature.com/articles/nature14541). If you want to know more about Bayesian neural networks (i.e. neural networks that learn a probability distribution over functions that are consistent with the data), you can read [this paper](http://proceedings.mlr.press/v37/blundell15.pdf).
  
  So, nowadays, I wouldn't say that that statement is "true", although it was probably true when that book was published. More and more, people in the machine learning community do research in PML and, in particular, BDL, because, if you want to adopt neural networks in areas like healthcare, you need to provide the doctors with some kind of uncertainty quantification of the predictions. Let's say that that a doctor needs to take an action, such as giving some kind of medicine to a patient based on the condition of the patient (e.g. temperature). The doctor doesn't just want to know "yes, give the medicine to the patient". The doctor wants to have an idea of how confident or uncertain the model is about its prediction. This is also where another subfield of AI comes into play, i.e. [*explainable AI*](https://ai.stackexchange.com/q/14224/2444), but I will not dwell more on this topic here.
  
  
  ---
  
  ### Answer by username_2 on 2021-08-25. Score: 0
  
  The confidence interval is a way of quantifying expected error in predictions.
  
  For example, lets say you are trying to model your dart throwing scoring metrics. If you play many games, and assume the score obtained follows a normal distribution, you can do a statistical calculation known as maximum likelihood estimation which provides you with parameters of the normal distribution that best fits the data. This allows you to quantify the error more precisely, so you can say things like: "95% of my games will be at least 200 points", or "having a score this good/bad is a 1 in 1000 chance."
  
  Much of machine learning is about outputting best predictions given some input, and not about quantifying error. So in the above example, we might just output a single value that best captures the underlying statistic, which in the above case, would be the mean of all scores, which is confirmed by the maximum likelihood estimate approach in statistics.
  
  A quick online search returns the definition: ["the probability that a population parameter will fall between a set of values for a certain proportion of times."](https://www.investopedia.com/terms/c/confidenceinterval.asp), which seems to agree with the above.
  
  Bar graphs with confidence intervals would be a very clear visual intuition of the idea.
  
  
  ---
  
  