Answers.Body: <h3>tl;dr</h3>
  <p>The safest method I've found is to use <strong>cross-validation</strong> for hyperparameter selection <strong>and a hold-out test set</strong> for a final evaluation.</p>
  <h3>Why this isn't working for you...</h3>
  <p>In your case, I suspect you're either running a <strong>large number of iterations during for hyperparameter selection</strong> or you have a fairly <strong>small dataset</strong> (or even a combination of both). If you can't find more data or use a larger dataset, I'd suggest limiting the <em>exhaustiveness</em> of your hyperparameter selection phase. If you run the process enough times the model is bound to overfit on the validation sets.</p>
  <p>Note that there is <strong>no guaranteed safe</strong> way of detecting overfitting before the test set!</p>
  <h3>Why I consider this strategy to be the safest</h3>
  <p>There are two different types of overfitting you need to be able to detect.</p>
  <p>The first is the most straightforward: <strong>overfitting on the training set</strong>. This means that the model has <strong>memorized</strong> the training set and can't generalize beyond that. If the test set even slightly differs from the training set (which is the case in most real-world problems), then the model will perform worse on it than on the training set. This is simple to detect and, in fact, the only thing you need to catch this is a test set!</p>
  <p>The second type of overfitting you need to detect is <strong>on the test set</strong>. Imagine you have a model and you make an exhaustive hyperparameter selection using a test set. Then you evaluate on the same test set. By doing this, you have adjusted your hyperparameters to achieve the best score <strong>for the specific test set</strong>. These hyperparameters are thus overfitting to that test set, even though the samples from that set were never seen during training. This is possible, because, during the iterative hyperparameter selection process, you have <strong>passed information about your test set to your model</strong>!</p>
  <p>This is much harder to detect. In fact, the only way to do so is to split the original data into three parts: the training, the validation and the test sets. The first is used for training the model, the second for hyperparameter selection, and the final is used <strong>only once</strong> for a final evaluation. If your model has overfitted (i.e. test performance is worse than training and validation), you need to <strong>start from scratch</strong>. Shuffle your data, split it again and repeat the process. This is usually referred to as a <a href="https://www.kdnuggets.com/wp-content/uploads/dataiku-holdout-strategy.jpg" rel="nofollow noreferrer">hold-out strategy</a>.</p>
  <p>To make the model even less prone to overfitting, you can use <a href="https://en.wikipedia.org/wiki/Cross-validation_(statistics)" rel="nofollow noreferrer">cross-validation</a> instead of a hold-out validation set. Because your model now has to be trained on <span class="math-container">$k$</span> slightly different training sets and evaluated on <span class="math-container">$k$</span> completely different validation sets, it is <strong>much harder</strong> for it to overfit on the validation sets (because it needs to <em>fool</em> <span class="math-container">$k$</span> different validation sets instead of one).</p>
  <h3>Cases where this might not be applicable</h3>
  <p>Depending on the circumstances, it might not be practical to apply both of these techniques.</p>
  <ul>
  <li><p>Cross-validation is pretty robust regarding overfitting but it imposes a <strong>computational burden</strong> to your process as it requires multiple trainings of the same model. This obviously isn't practical for computationally expensive models (e.g. image classifiers). In this case, use a hold-out strategy as mentioned previously.</p>
  </li>
  <li><p>Using a hold-out test set means that you are reducing the size of your training set, which might actually make your model <strong>more prone to overfitting</strong> (i.e. have a higher-variance) for <strong>small datasets</strong>. In this case (if your model is practically untrainable due to the small size) you can resort to cross-validation, <strong>but you risk overfitting on the validation set</strong> and not having any way to detect it.</p>
  </li>
  </ul>
  <h3>How to combat overfitting</h3>
  <p>Since it is fairly related, I'll post a link to an <a href="https://stats.stackexchange.com/a/365806/119015">answer on how to combat overfitting</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-06T00:02:22.873Z
Answers.Text: In addition to what you wrote, I think it's important to mention that if you have the possibility to shuffle your whole dataset before splitting it into training, validation, and test sets, it's probably a good idea to do that, so that the distributions of these datasets are similar, which is an assumption that most algorithms make. If you don't shuffle the dataset before splitting, you could end up with training, validation and test datasets that have quite different distributions, so it's normal that the model doesn't perform well on the test set.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-31T21:47:49.937Z
Answers.LastActivityDate: 2021-06-06T00:00:13.553Z
Answers.LastEditDate: 2021-06-06T00:00:13.553Z
Answers.UserDisplayName: Djib2011
Body: <p>I wanted to ask about the methodology of testing the ML models against overfitting. Please note that I don't mean any overfitting reducing methods like regularisation, just a measure to judge whether a model has overfitting problems. </p>
  
  <p>I am currently developing a framework for tuning models (features, hyperparameters) based on evolutionary algorithms. And the problem that I face is the lack of a good method to judge if the model overfits before using the test set. I encountered the cases where the model that was good on both training and validation sets, behaved poorly on the test set for both randomized and not randomized training and validation splits. I used k-fold cross-validation with additionally estimating the standard deviation of all folds results (the smaller deviation means better model), but, still, it doesn't work as expected. </p>
  
  <p>Summing up, I usually don't see a correlation (or a very poor one) between training, validation and k-fold errors with test errors. In other words, tuning the model to obtain lower values of any of the above mentioned measures usually does not mean lowering the test error.</p>
  
  <p>Could I ask you, how in practice you test your models? And maybe there are some new methods not mentioned in typical ML books?</p>
  
Tags: <machine-learning><overfitting><cross-validation><loocv><k-fold-cv>
Title: What is the best measure for detecting overfitting?
UserDisplayName: GKozinski
ThreadText: # What is the best measure for detecting overfitting?
  
  (Asked by: username_0 on 2019-07-31)
  
  I wanted to ask about the methodology of testing the ML models against overfitting. Please note that I don't mean any overfitting reducing methods like regularisation, just a measure to judge whether a model has overfitting problems.
  
  I am currently developing a framework for tuning models (features, hyperparameters) based on evolutionary algorithms. And the problem that I face is the lack of a good method to judge if the model overfits before using the test set. I encountered the cases where the model that was good on both training and validation sets, behaved poorly on the test set for both randomized and not randomized training and validation splits. I used k-fold cross-validation with additionally estimating the standard deviation of all folds results (the smaller deviation means better model), but, still, it doesn't work as expected.
  
  Summing up, I usually don't see a correlation (or a very poor one) between training, validation and k-fold errors with test errors. In other words, tuning the model to obtain lower values of any of the above mentioned measures usually does not mean lowering the test error.
  
  Could I ask you, how in practice you test your models? And maybe there are some new methods not mentioned in typical ML books?
  
  **Tags:** machine-learning, overfitting, cross-validation, loocv, k-fold-cv
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-07-31. Score: 5 | Accepted answer
  
  ### tl;dr
  
  The safest method I've found is to use **cross-validation** for hyperparameter selection **and a hold-out test set** for a final evaluation.
  
  ### Why this isn't working for you...
  
  In your case, I suspect you're either running a **large number of iterations during for hyperparameter selection** or you have a fairly **small dataset** (or even a combination of both). If you can't find more data or use a larger dataset, I'd suggest limiting the *exhaustiveness* of your hyperparameter selection phase. If you run the process enough times the model is bound to overfit on the validation sets.
  
  Note that there is **no guaranteed safe** way of detecting overfitting before the test set!
  
  ### Why I consider this strategy to be the safest
  
  There are two different types of overfitting you need to be able to detect.
  
  The first is the most straightforward: **overfitting on the training set**. This means that the model has **memorized** the training set and can't generalize beyond that. If the test set even slightly differs from the training set (which is the case in most real-world problems), then the model will perform worse on it than on the training set. This is simple to detect and, in fact, the only thing you need to catch this is a test set!
  
  The second type of overfitting you need to detect is **on the test set**. Imagine you have a model and you make an exhaustive hyperparameter selection using a test set. Then you evaluate on the same test set. By doing this, you have adjusted your hyperparameters to achieve the best score **for the specific test set**. These hyperparameters are thus overfitting to that test set, even though the samples from that set were never seen during training. This is possible, because, during the iterative hyperparameter selection process, you have **passed information about your test set to your model**!
  
  This is much harder to detect. In fact, the only way to do so is to split the original data into three parts: the training, the validation and the test sets. The first is used for training the model, the second for hyperparameter selection, and the final is used **only once** for a final evaluation. If your model has overfitted (i.e. test performance is worse than training and validation), you need to **start from scratch**. Shuffle your data, split it again and repeat the process. This is usually referred to as a [hold-out strategy](https://www.kdnuggets.com/wp-content/uploads/dataiku-holdout-strategy.jpg).
  
  To make the model even less prone to overfitting, you can use [cross-validation](https://en.wikipedia.org/wiki/Cross-validation_(statistics)) instead of a hold-out validation set. Because your model now has to be trained on $k$ slightly different training sets and evaluated on $k$ completely different validation sets, it is **much harder** for it to overfit on the validation sets (because it needs to *fool* $k$ different validation sets instead of one).
  
  ### Cases where this might not be applicable
  
  Depending on the circumstances, it might not be practical to apply both of these techniques.
  
  * Cross-validation is pretty robust regarding overfitting but it imposes a **computational burden** to your process as it requires multiple trainings of the same model. This obviously isn't practical for computationally expensive models (e.g. image classifiers). In this case, use a hold-out strategy as mentioned previously.
  * Using a hold-out test set means that you are reducing the size of your training set, which might actually make your model **more prone to overfitting** (i.e. have a higher-variance) for **small datasets**. In this case (if your model is practically untrainable due to the small size) you can resort to cross-validation, **but you risk overfitting on the validation set** and not having any way to detect it.
  
  ### How to combat overfitting
  
  Since it is fairly related, I'll post a link to an [answer on how to combat overfitting](https://stats.stackexchange.com/a/365806/119015).
  
  > **Comments:**
  > **username_2**: In addition to what you wrote, I think it's important to mention that if you have the possibility to shuffle your whole dataset before splitting it into training, validation, and test sets, it's probably a good idea to do that, so that the distributions of these datasets are similar, which is an assumption that most algorithms make. If you don't shuffle the dataset before splitting, you could end up with training, validation and test datasets that have quite different distributions, so it's normal that the model doesn't perform well on the test set.  
  
  ---
  
  