Answers.Body: <p>The issue is the generalisation. If your model fits perfectly to your training data, it then depends whether your training data is a good reflection of the actual data you will encounter 'in the wild'. If it was a perfect sample, and no other values ever occur, then there is no problem.</p>
  <p>But, as is more likely, the training data is only a small and not 100% representative sample. Now your model fits this data perfectly, but previously unseen (noisy) data is different, and doesn't fit, because your model is too specific.</p>
  <p>With classification you want to allow for some variability, because real-world data is never perfect. Your accuracy (of classifying items) might improve if your model is more forgiving (ie less specific).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-27T20:10:10.450Z
Answers.Text: But the model is fitting to everything, the noise and also the actual data that it should be learning. So why does the noise ruin the generalization?
Answers.UserDisplayName: JobHunter69
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-28T10:56:18.463Z
Answers.Text: @JobHunter69 The noise will be random, and thus will mask the 'actual' (idealised) data. But you don't know what is noise and what is the data, so the noise will be treated as part of the data and reduce accuracy. With more data, this effect diminishes.
Answers.UserDisplayName: Oliver Mason
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-28T15:45:40.420Z
Answers.Text: If the model fits both to the noise and also the data, why does the noise mask the data? The model should still be able to learn the data
Answers.UserDisplayName: JobHunter69
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-29T09:06:26.587Z
Answers.Text: The model cannot distinguish between the two.
Answers.UserDisplayName: Oliver Mason
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-03T09:40:08.400Z
Answers.LastActivityDate: 2024-01-03T09:40:08.400Z
Answers.UserDisplayName: Oliver Mason
Answers.Body: <p>Note that if you know the function your data was generated from, you do not even need machine learning. For example, if you generated the data with <span class="math-container">$y = f(x) = x^2$</span>, you already have the function to predict future <span class="math-container">$y$</span>s given the <span class="math-container">$x$</span>s, i.e. <span class="math-container">$f(x) = x^2$</span>.</p>
  <p>So, we usually apply machine learning when we don't know the target function, i.e. when you don't know the shape of some function you want to learn, although you can make assumptions about it, which may be wrong. For example, you could assume your function is a parabola <span class="math-container">$x^2$</span>, but it could also be any other function that may have data in common with <span class="math-container">$x^2$</span>. So, if your limited training data suggests that the function is a parabola, but in reality is not, you will learn a parabola (overfit), instead of learning the actual target function (i.e. generalise). It can be a big problem if the target function is actually very different from the function you learned, which may be the case if you used small or non-representative training dataset, because your model will give wrong predictions for unseen data.</p>
  <p>So, you almost never want to overfit.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-03T11:29:14.880Z
Answers.Text: Oliver's answer is correct. I'm just giving a slightly different answer.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-03T11:21:41.463Z
Answers.LastActivityDate: 2024-01-03T11:28:48.540Z
Answers.LastEditDate: 2024-01-03T11:28:48.540Z
Answers.UserDisplayName: nbro
Body: <p>If a model overfit to the training data, why does it generalize poorly?</p>
  <p>Consider the basic problem of a noisy 2d dataset where I am fitting polynomials. A good model would be a parabola and a line would be underfitting. As I increase the polynomial's power, I end up overfitting more and more to the data. However, why is the data unlikely to follow this overfitting behavior? The only knowledge I have is of this data. It would be reasonable to assume the better the model does on this data, the better it would do on testing dataset.</p>
  <p>How do I increase the complexity of my model or training data such that accuracy can only be improved? For example if I have a given model that does generalizes well, how do I keep this good generalization while it fits better to the training data?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-01-03T11:12:11.147Z
Comments.Text: I think you're asking different questions here. The answer below answers to your question in the title. Please, consider removing your other questions from this post and ask them in separate posts, if you want specific answers to them. Thank you.
Comments.UserDisplayName: nbro
Tags: <machine-learning><overfitting><statistics><generalization>
Title: Why does model overfitting lead to poor generalization?
UserDisplayName: JobHunter69
ThreadText: # Why does model overfitting lead to poor generalization?
  
  (Asked by: username_0 on 2024-01-02)
  
  If a model overfit to the training data, why does it generalize poorly?
  
  Consider the basic problem of a noisy 2d dataset where I am fitting polynomials. A good model would be a parabola and a line would be underfitting. As I increase the polynomial's power, I end up overfitting more and more to the data. However, why is the data unlikely to follow this overfitting behavior? The only knowledge I have is of this data. It would be reasonable to assume the better the model does on this data, the better it would do on testing dataset.
  
  How do I increase the complexity of my model or training data such that accuracy can only be improved? For example if I have a given model that does generalizes well, how do I keep this good generalization while it fits better to the training data?
  
  **Tags:** machine-learning, overfitting, statistics, generalization
  
  > **Comments:**
  > **username_1**: I think you're asking different questions here. The answer below answers to your question in the title. Please, consider removing your other questions from this post and ask them in separate posts, if you want specific answers to them. Thank you.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2024-01-03. Score: 3
  
  The issue is the generalisation. If your model fits perfectly to your training data, it then depends whether your training data is a good reflection of the actual data you will encounter 'in the wild'. If it was a perfect sample, and no other values ever occur, then there is no problem.
  
  But, as is more likely, the training data is only a small and not 100% representative sample. Now your model fits this data perfectly, but previously unseen (noisy) data is different, and doesn't fit, because your model is too specific.
  
  With classification you want to allow for some variability, because real-world data is never perfect. Your accuracy (of classifying items) might improve if your model is more forgiving (ie less specific).
  
  > **Comments:**
  > **username_0**: But the model is fitting to everything, the noise and also the actual data that it should be learning. So why does the noise ruin the generalization?  
  > **username_2**: @username_0 The noise will be random, and thus will mask the 'actual' (idealised) data. But you don't know what is noise and what is the data, so the noise will be treated as part of the data and reduce accuracy. With more data, this effect diminishes.  
  > **username_0**: If the model fits both to the noise and also the data, why does the noise mask the data? The model should still be able to learn the data  
  > **username_2**: The model cannot distinguish between the two.  
  
  ---
  
  ### Answer by username_1 on 2024-01-03. Score: 2
  
  Note that if you know the function your data was generated from, you do not even need machine learning. For example, if you generated the data with $y = f(x) = x^2$, you already have the function to predict future $y$s given the $x$s, i.e. $f(x) = x^2$.
  
  So, we usually apply machine learning when we don't know the target function, i.e. when you don't know the shape of some function you want to learn, although you can make assumptions about it, which may be wrong. For example, you could assume your function is a parabola $x^2$, but it could also be any other function that may have data in common with $x^2$. So, if your limited training data suggests that the function is a parabola, but in reality is not, you will learn a parabola (overfit), instead of learning the actual target function (i.e. generalise). It can be a big problem if the target function is actually very different from the function you learned, which may be the case if you used small or non-representative training dataset, because your model will give wrong predictions for unseen data.
  
  So, you almost never want to overfit.
  
  > **Comments:**
  > **username_1**: username_2's answer is correct. I'm just giving a slightly different answer.  
  
  ---
  
  