Answers.Body: <p>Yes, precisely.</p>
  <p>What do you mention is known in the literature as the <strong>Bayes Error</strong>. See page top of the page 114 <a href="https://www.deeplearningbook.org/contents/ml.html" rel="nofollow noreferrer">https://www.deeplearningbook.org/contents/ml.html</a>.</p>
  <blockquote>
  <p>The ideal model is an oracle that simply knows the true probability distribution that generates the data. Even such a model will still incur some error on many problems, because there may still be some noise in the distribution. In the case of supervised learning, the mapping from <span class="math-container">$\mathbf{x}$</span> to <span class="math-container">$y$</span> may be inherently stochastic, or <span class="math-container">$y$</span> may be a deterministic function that involves other variables besides those included in <span class="math-container">$\mathbf{x}$</span>. The error incurred by an oracle making predictions from the true distribution <span class="math-container">$p(\mathbf{x}, y)$</span> is called the <strong>Bayes error</strong>.</p>
  </blockquote>
  <p>Regardless of how clever is your model, the best error you can achieve for the prediction on the data distribution is <span class="math-container">$\varepsilon$</span>. Note that it holds for the whole data distribution, not a sample of data.</p>
  <p>Say, you would like to fit something like <span class="math-container">$\sin(x) + \varepsilon$</span>. There are 10 points, and one can fit them perfectly with the 9th-degree polynomial, but this an error on training data, and, in case one samples more data points, the error will be likely do exceed the optimal <span class="math-container">$\varepsilon$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-30T14:09:28.683Z
Answers.LastActivityDate: 2021-09-30T16:22:52.403Z
Answers.LastEditDate: 2021-09-30T16:22:52.403Z
Answers.UserDisplayName: spiridon_the_sun_rotator
Body: <p>In supervised machine learning, it is common to say that we learn a function of the form</p>
  <p><span class="math-container">$$y=g(x) + \epsilon.$$</span></p>
  <p>Generally, <span class="math-container">$\epsilon$</span> is used to denote noise or, more precisely, any influence by latent variables such as measurement inaccuracies (right?).</p>
  <p>Is it, therefore, correct to say that we use <span class="math-container">$\epsilon$</span> to denote the model's imperfection to the real world (caused by anything unknown)?</p>
  
Tags: <supervised-learning><noise>
Title: Is the noise term $\epsilon$ in $y=g(x) + \epsilon$ used to denote the model's imperfection to the real world?
UserDisplayName: Micha Christ
ThreadText: # Is the noise term $\epsilon$ in $y=g(x) + \epsilon$ used to denote the model's imperfection to the real world?
  
  (Asked by: username_0 on 2021-09-30)
  
  In supervised machine learning, it is common to say that we learn a function of the form
  
  $$y=g(x) + \epsilon.$$
  
  Generally, $\epsilon$ is used to denote noise or, more precisely, any influence by latent variables such as measurement inaccuracies (right?).
  
  Is it, therefore, correct to say that we use $\epsilon$ to denote the model's imperfection to the real world (caused by anything unknown)?
  
  **Tags:** supervised-learning, noise
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-09-30. Score: 2
  
  Yes, precisely.
  
  What do you mention is known in the literature as the **Bayes Error**. See page top of the page 114 <https://www.deeplearningbook.org/contents/ml.html>.
  
  > The ideal model is an oracle that simply knows the true probability distribution that generates the data. Even such a model will still incur some error on many problems, because there may still be some noise in the distribution. In the case of supervised learning, the mapping from $\mathbf{x}$ to $y$ may be inherently stochastic, or $y$ may be a deterministic function that involves other variables besides those included in $\mathbf{x}$. The error incurred by an oracle making predictions from the true distribution $p(\mathbf{x}, y)$ is called the **Bayes error**.
  
  Regardless of how clever is your model, the best error you can achieve for the prediction on the data distribution is $\varepsilon$. Note that it holds for the whole data distribution, not a sample of data.
  
  Say, you would like to fit something like $\sin(x) + \varepsilon$. There are 10 points, and one can fit them perfectly with the 9th-degree polynomial, but this an error on training data, and, in case one samples more data points, the error will be likely do exceed the optimal $\varepsilon$.
  
  
  ---
  
  