Answers.Body: <blockquote>
    <p>Is overfitting always a bad thing?</p>
  </blockquote>
  
  <p>The answer is a resounding <strong>yes</strong>, every time. The reason being that overfitting is the name we use to refer to a situation where your model did very well on the training data but when you showed it the dataset that really matter(i.e the test data or put it into production), it performed very bad. This can never be good, because we write models so that they can predict with high accuracy on the datasets it has not seen before. No one cares about a model that got 100% accuracy on the training set and gets 0% on the test set. This is a useless model. We train models so that they can predict with high accuracy on data it has never seen before.</p>
  
  <p><strong>Now for the long answer:</strong></p>
  
  <p>I will start by giving a quick explanation of what overfitting is. Then how we can detect overfitting, this will help me explain why overfitting is <strong>allways</strong> a bad.  </p>
  
  <p><strong>What is overfitting:</strong></p>
  
  <p>In machine learning problem there is signal then there is noise. Signal is the true underlying pattern that you wish to learn from the data whist noise is the irrelevant information or randomness in a dataset.</p>
  
  <p>The reason that leads to overfitting is that your model is making use of "Noise" to make decisions instead of using the signal. Overfitting means that your model performs very well on the training data and performs very poorly on unseen dataset(s) (notice that we pretty much don't care how well a model performs on the training data, we care about how it will perform on data it hasn't seen before). A model that has overfitted is similar to a student who got 100% on the quiz he/she did in preparation for the exam and will get a very poor mark on the exam because he/she because he chose to cram/memorize the quiz's answers and questions instead of understanding how to get to those answers. Another way of thinking about this is that your model has memorized/crammed instead of learning the learning the general relationship contained in the training dataset. </p>
  
  <p><strong>How do we detect Overfitting:</strong></p>
  
  <p>The only way to know if your model is overfitting is to test its performance on a dataset it has never seen before. As such, overfitting is something you know with the benefit of hindsight following testing a model on data it has never seen before.  </p>
  
  <blockquote>
    <p>For this usage, where the answer is much simpler than recognizing a cat, isn't overfitting a good thing?</p>
  </blockquote>
  
  <p>No, because if your model is overfitting it will perform bad on the test set(in your case, it will perform very bad when you deploy it in a real world ACAS and start showing it data it hasnt seen before). see <em>How do we detect Overfitting</em> above.   </p>
  
  <p>If it turns out that your model performs very well when you show it data it hasn't seen before, then that means it didn't overfit (see <em>What is overfitting</em> above). </p>
  
  <blockquote>
    <p>What would overfitting "look like" in this case and why would it be bad ?
    Your model will get very high accuracy on the training set and get very poor accuracy when you start showing it examples it hasn't seen before. So in short, you will get your hopes high, jump up and down in excitement thinking that your model is great only for it no get very poor results in a production ACAS system.  </p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-14T06:28:44.310Z
Answers.Text: Let's say a model has 100% accuracy on training data, and 80% accuracy on data it has never seen before; let's say that 80% is a good result for this type of problem. Would you say the model is overfitted?
Answers.UserDisplayName: Akavall
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-18T02:17:08.193Z
Answers.Text: Bump on Akavall's comment. 
  
  I'd say by definition, the model is overfitted. However, if 80% is a good result for that problem, I don't see why that would be a bad model. Perhaps calculate additional metrics to see a more in-depth view of the model.
Answers.UserDisplayName: LeggoMaEggo
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-09-30T19:10:46.590Z
Answers.LastActivityDate: 2017-09-30T19:10:46.590Z
Answers.UserDisplayName: Tshilidzi Mudau
Answers.Body: <p><em>Overfitting</em> is almost always bad and hurts <em>generalization</em>. You say</p>
  
  <blockquote>
    <p>what we want from the NN is to say "take a slight left turn" or "turn
    right hard" if another ship comes slightly close on the right or
    rapidly close on the left, respectively.</p>
  </blockquote>
  
  <p>But what would you say if the NN learns to "take a slight left turn" only if the coming ship is small (because accidentally that's what the training set includes)? Or it learns to "turn right hard" only on a certain altitude and gives arbitrary answer when the altitude is different?</p>
  
  <p>In practice, overfitting means <em>learning the noise</em>, i.e. the patterns that are not there. I can hardly imagine a good example, when it's what you really need.</p>
  
  <p>To answer you question: it's quite possible that for ACAS task a much smaller and simpler model would work fine, not necessarily deep and even not necessarily neural network. But it must be reasonable and learning the white noise is not what you call reasonable.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-09-30T17:32:08.057Z
Answers.LastActivityDate: 2017-09-30T17:32:08.057Z
Answers.UserDisplayName: Maxim
Answers.Body: <p>Typically the ramification of overfitting is poor performance on unseen data. If you're confident that overfitting on your dataset will not cause problems for situations not described by the dataset, or the dataset contains every possible scenario then overfitting may be good for the performance of the NN.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-09-28T20:12:55.737Z
Answers.LastActivityDate: 2017-09-28T20:12:55.737Z
Answers.UserDisplayName: user
Answers.Body: <p>No. When a system where input and output is well known ie. the state space is fully known and defined using say physics (PDEs) then overfitting is desirable as there is no need to generalize. An ML model has low inference time compared to solving the PDEs or lookup table is of multiple terabytes.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-16T05:55:56.967Z
Answers.Text: With such complete training data, is the result "overfitting" or just "fitting" though? I would say with such training data it is very hard to overfit, but overfitting could still be bad. You could get still get non-generalising behaviour with poor quality results between points at *any* density of training data, at least theoretically.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-16T07:01:42.130Z
Answers.Text: Nope. There is no need to generalize here. You have the complete state space.
Answers.UserDisplayName: kosmos
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-16T09:35:34.100Z
Answers.Text: If you have the complete state space, then arguably you are not overfitting. You are fitting to the data. So it is just semantics to say "overfitting is desirable". It is like defining a speed limit to be infinite by law and saying "speeding is desirable". IMO meaningless word play
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-16T10:11:02.567Z
Answers.Text: @NeilSlater in terms of machine learning it is overfitting and a very useful term to use here. As it will direct one to not be using regularization, dropout, validation test etc. It is neither meaningless nor wordplay. This is called "useful overfitting". I suggest you be cautious while forming opinions at high speed giving little food for thought.
Answers.UserDisplayName: kosmos
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-16T13:21:06.133Z
Answers.Text: OK, so if it is overfitting, how does it manifest, exactly? Typically you will see validation error rise whilst training error falls. But that cannot happen when your training data is the entire function domain. How do you know your training has overfit, in order to say that you have "useful overfitting"?
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-16T13:45:18.920Z
Answers.Text: I am not saying that having a lot of data, therefore not needing to regularise, is a bad thing, or that such scenarios don't happen. I am questioning the purpose of calling that "overfitting". Instead it appears to be training with enough data that overfitting is no longer a concern. It's not that overfitting is happening and somehow good now. It's that overfitting is *not* happening.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-16T02:33:14.690Z
Answers.LastActivityDate: 2021-09-16T02:33:14.690Z
Answers.UserDisplayName: kosmos
Body: <p>DNN can be used to recognize pictures. Great. For that usage, it's better if they are somewhat flexible so as to recognize as cats even cats that are not on the pictures on which they trained (i.e. avoid overfitting). Agreed.</p>
  
  <p>But when <a href="http://ieeexplore.ieee.org/abstract/document/7778091/" rel="noreferrer">one uses</a> NN as a replacement for numerical tables in an Air Collision Avoidance System (ACAS), it is primarily to reduce the "required storage space by a factor of 1000". For this usage, what we want from the NN is to say "take a slight left turn" or "turn right hard" if another ship comes slightly close on the right or rapidly close on the left, respectively. </p>
  
  <p>For this usage, where the answer is much simpler than recognizing a cat, isn't overfitting a good thing? What would overfitting "look like" in this case and why would it be bad ?</p>
  
  <p>This question somewhat relates <a href="http://ai.stackexchange.com/questions/3473/is-there-such-a-thing-like-the-machine-learning-paradox">to this one</a>, where a general idea seems to be "Machine Learning is used for intractable things, you don't need ML for tractable things". And while it is quite correct that ACAS can be implemented without NN, I wouldn't call NN "useless" for ACAS, because a factor 1000 reduction in required space will always come in handy.  </p>
  
Tags: <neural-networks><control-problem><overfitting>
Title: Is overfitting always a bad thing?
UserDisplayName: ZakC
ThreadText: # Is overfitting always a bad thing?
  
  (Asked by: username_0 on 2017-09-28)
  
  DNN can be used to recognize pictures. Great. For that usage, it's better if they are somewhat flexible so as to recognize as cats even cats that are not on the pictures on which they trained (i.e. avoid overfitting). Agreed.
  
  But when [one uses](http://ieeexplore.ieee.org/abstract/document/7778091/) NN as a replacement for numerical tables in an Air Collision Avoidance System (ACAS), it is primarily to reduce the "required storage space by a factor of 1000". For this usage, what we want from the NN is to say "take a slight left turn" or "turn right hard" if another ship comes slightly close on the right or rapidly close on the left, respectively.
  
  For this usage, where the answer is much simpler than recognizing a cat, isn't overfitting a good thing? What would overfitting "look like" in this case and why would it be bad ?
  
  This question somewhat relates [to this one](http://ai.stackexchange.com/questions/3473/is-there-such-a-thing-like-the-machine-learning-paradox), where a general idea seems to be "Machine Learning is used for intractable things, you don't need ML for tractable things". And while it is quite correct that ACAS can be implemented without NN, I wouldn't call NN "useless" for ACAS, because a factor 1000 reduction in required space will always come in handy.
  
  **Tags:** neural-networks, control-problem, overfitting
  
  ---
  
  ## 4 answers
  
  ### Answer by username_5 on 2017-09-28. Score: 7
  
  Typically the ramification of overfitting is poor performance on unseen data. If you're confident that overfitting on your dataset will not cause problems for situations not described by the dataset, or the dataset contains every possible scenario then overfitting may be good for the performance of the NN.
  
  
  ---
  
  ### Answer by username_1 on 2017-09-30. Score: 5
  
  > Is overfitting always a bad thing?
  
  The answer is a resounding **yes**, every time. The reason being that overfitting is the name we use to refer to a situation where your model did very well on the training data but when you showed it the dataset that really matter(i.e the test data or put it into production), it performed very bad. This can never be good, because we write models so that they can predict with high accuracy on the datasets it has not seen before. No one cares about a model that got 100% accuracy on the training set and gets 0% on the test set. This is a useless model. We train models so that they can predict with high accuracy on data it has never seen before.
  
  **Now for the long answer:**
  
  I will start by giving a quick explanation of what overfitting is. Then how we can detect overfitting, this will help me explain why overfitting is **allways** a bad.
  
  **What is overfitting:**
  
  In machine learning problem there is signal then there is noise. Signal is the true underlying pattern that you wish to learn from the data whist noise is the irrelevant information or randomness in a dataset.
  
  The reason that leads to overfitting is that your model is making use of "Noise" to make decisions instead of using the signal. Overfitting means that your model performs very well on the training data and performs very poorly on unseen dataset(s) (notice that we pretty much don't care how well a model performs on the training data, we care about how it will perform on data it hasn't seen before). A model that has overfitted is similar to a student who got 100% on the quiz he/she did in preparation for the exam and will get a very poor mark on the exam because he/she because he chose to cram/memorize the quiz's answers and questions instead of understanding how to get to those answers. Another way of thinking about this is that your model has memorized/crammed instead of learning the learning the general relationship contained in the training dataset.
  
  **How do we detect Overfitting:**
  
  The only way to know if your model is overfitting is to test its performance on a dataset it has never seen before. As such, overfitting is something you know with the benefit of hindsight following testing a model on data it has never seen before.
  
  > For this usage, where the answer is much simpler than recognizing a cat, isn't overfitting a good thing?
  
  No, because if your model is overfitting it will perform bad on the test set(in your case, it will perform very bad when you deploy it in a real world ACAS and start showing it data it hasnt seen before). see *How do we detect Overfitting* above.
  
  If it turns out that your model performs very well when you show it data it hasn't seen before, then that means it didn't overfit (see *What is overfitting* above).
  
  > What would overfitting "look like" in this case and why would it be bad ?
  > Your model will get very high accuracy on the training set and get very poor accuracy when you start showing it examples it hasn't seen before. So in short, you will get your hopes high, jump up and down in excitement thinking that your model is great only for it no get very poor results in a production ACAS system.
  
  > **Comments:**
  > **username_2**: Let's say a model has 100% accuracy on training data, and 80% accuracy on data it has never seen before; let's say that 80% is a good result for this type of problem. Would you say the model is overfitted?  
  > **username_3**: Bump on Akavall's comment.   I'd say by definition, the model is overfitted. However, if 80% is a good result for that problem, I don't see why that would be a bad model. Perhaps calculate additional metrics to see a more in-depth view of the model.  
  
  ---
  
  ### Answer by username_4 on 2017-09-30. Score: 3
  
  *Overfitting* is almost always bad and hurts *generalization*. You say
  
  > what we want from the NN is to say "take a slight left turn" or "turn
  > right hard" if another ship comes slightly close on the right or
  > rapidly close on the left, respectively.
  
  But what would you say if the NN learns to "take a slight left turn" only if the coming ship is small (because accidentally that's what the training set includes)? Or it learns to "turn right hard" only on a certain altitude and gives arbitrary answer when the altitude is different?
  
  In practice, overfitting means *learning the noise*, i.e. the patterns that are not there. I can hardly imagine a good example, when it's what you really need.
  
  To answer you question: it's quite possible that for ACAS task a much smaller and simpler model would work fine, not necessarily deep and even not necessarily neural network. But it must be reasonable and learning the white noise is not what you call reasonable.
  
  
  ---
  
  ### Answer by username_6 on 2021-09-16. Score: 1
  
  No. When a system where input and output is well known ie. the state space is fully known and defined using say physics (PDEs) then overfitting is desirable as there is no need to generalize. An ML model has low inference time compared to solving the PDEs or lookup table is of multiple terabytes.
  
  > **Comments:**
  > **username_7**: With such complete training data, is the result "overfitting" or just "fitting" though? I would say with such training data it is very hard to overfit, but overfitting could still be bad. You could get still get non-generalising behaviour with poor quality results between points at *any* density of training data, at least theoretically.  
  > **username_6**: Nope. There is no need to generalize here. You have the complete state space.  
  > **username_7**: If you have the complete state space, then arguably you are not overfitting. You are fitting to the data. So it is just semantics to say "overfitting is desirable". It is like defining a speed limit to be infinite by law and saying "speeding is desirable". IMO meaningless word play  
  > **username_6**: @NeilSlater in terms of machine learning it is overfitting and a very useful term to use here. As it will direct one to not be using regularization, dropout, validation test etc. It is neither meaningless nor wordplay. This is called "useful overfitting". I suggest you be cautious while forming opinions at high speed giving little food for thought.  
  > **username_7**: OK, so if it is overfitting, how does it manifest, exactly? Typically you will see validation error rise whilst training error falls. But that cannot happen when your training data is the entire function domain. How do you know your training has overfit, in order to say that you have "useful overfitting"?  
  > **username_7**: I am not saying that having a lot of data, therefore not needing to regularise, is a bad thing, or that such scenarios don't happen. I am questioning the purpose of calling that "overfitting". Instead it appears to be training with enough data that overfitting is no longer a concern. It's not that overfitting is happening and somehow good now. It's that overfitting is *not* happening.  
  
  ---
  
  