Answers.Body: <p>Error Estimation is a subject with a long history. The test-set method is only one way to estimate generalization error. Others include resubstitution, cross-validation, bootstrap, posterior-probability estimators, and bolstered estimators. These and more are reviewed, for instance, in the book: Braga-Neto and Dougherty, &quot;Error Estimation for Pattern Recognition,&quot; IEEE-Wiley, 2015.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-16T23:41:29.197Z
Answers.LastActivityDate: 2020-08-16T23:41:29.197Z
Answers.UserDisplayName: Ulisses Braga-Neto
Answers.Body: <p>It's basically not possible to test besides some empirical experiments. All the generalization bounds only apply if your process actually follows the model assumptions which you don't actually know to be true.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-07T17:39:03.453Z
Answers.Text: This is wrong. The generalization error can be [**estimated**](https://www.merriam-webster.com/thesaurus/estimated). [This answer](https://ai.stackexchange.com/a/138/2444) provides a way to **estimate** the generalization error. Now, if you're talking about generalization bounds, that's something different. However, the question was about **estimating** the generalization error.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-07T20:37:14.093Z
Answers.Text: You don't know if your validation data actually comes from the same distribution or.not basically it's all guess, you never know much of anything until you run the system real time.
Answers.UserDisplayName: FourierFlux
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-07T17:29:59.017Z
Answers.LastActivityDate: 2020-11-08T13:07:02.920Z
Answers.LastEditDate: 2020-11-08T13:07:02.920Z
Answers.UserDisplayName: FourierFlux
Answers.Body: <p>Generalization error is the error obtained by applying a model to data it has not seen before. So, if you want to measure generalization error, you need to remove a subset from your data and don't train your model on it. After training, you verify your model accuracy (or other performance measures) on the subset you have removed since your model hasn't seen it before. Hence, this subset is called a <a href="https://en.wikipedia.org/wiki/Test_set" rel="nofollow noreferrer">test set</a>. </p>
  
  <p>Additionally, another subset can also be used for parameter selection, which we call a <a href="https://en.wikipedia.org/wiki/Test_set#Validation_set" rel="nofollow noreferrer">validation set</a>. We can't use the training set for parameter tuning, since it does not measure generalization error, but we can't use the test set too since our parameter tuning would overfit test data. That's why we need a third subset.</p>
  
  <p>Finally, in order to obtain more predictive performance measures, we can use many different train/test partitions and average the results. This is called <a href="https://en.wikipedia.org/wiki/Cross-validation_(statistics)" rel="nofollow noreferrer">cross-validation</a>.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-02T20:32:12.480Z
Answers.LastActivityDate: 2018-04-11T23:42:57.393Z
Answers.LastEditDate: 2018-04-11T23:42:57.393Z
Answers.UserDisplayName: rcpinto
Body: <p>How would you estimate the generalization error? What are the methods of achieving this?</p>
  
Tags: <machine-learning><deep-neural-networks><overfitting><computational-learning-theory><generalization>
Title: How can the generalization error be estimated?
UserDisplayName: kenorb
ThreadText: # How can the generalization error be estimated?
  
  (Asked by: username_0 on 2016-08-02)
  
  How would you estimate the generalization error? What are the methods of achieving this?
  
  **Tags:** machine-learning, deep-neural-networks, overfitting, computational-learning-theory, generalization
  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_4 on 2016-08-02. Score: 6 | Accepted answer
  
  Generalization error is the error obtained by applying a model to data it has not seen before. So, if you want to measure generalization error, you need to remove a subset from your data and don't train your model on it. After training, you verify your model accuracy (or other performance measures) on the subset you have removed since your model hasn't seen it before. Hence, this subset is called a [test set](https://en.wikipedia.org/wiki/Test_set).
  
  Additionally, another subset can also be used for parameter selection, which we call a [validation set](https://en.wikipedia.org/wiki/Test_set#Validation_set). We can't use the training set for parameter tuning, since it does not measure generalization error, but we can't use the test set too since our parameter tuning would overfit test data. That's why we need a third subset.
  
  Finally, in order to obtain more predictive performance measures, we can use many different train/test partitions and average the results. This is called [cross-validation](https://en.wikipedia.org/wiki/Cross-validation_(statistics)).
  
  
  ---
  
  ### Answer by username_1 on 2020-08-16. Score: 1
  
  Error Estimation is a subject with a long history. The test-set method is only one way to estimate generalization error. Others include resubstitution, cross-validation, bootstrap, posterior-probability estimators, and bolstered estimators. These and more are reviewed, for instance, in the book: username_1 and Dougherty, "Error Estimation for Pattern Recognition," IEEE-Wiley, 2015.
  
  
  ---
  
  ### Answer by username_2 on 2020-11-07. Score: -1
  
  It's basically not possible to test besides some empirical experiments. All the generalization bounds only apply if your process actually follows the model assumptions which you don't actually know to be true.
  
  > **Comments:**
  > **username_3**: This is wrong. The generalization error can be [**estimated**](https://www.merriam-webster.com/thesaurus/estimated). [This answer](https://ai.stackexchange.com/a/138/2444) provides a way to **estimate** the generalization error. Now, if you're talking about generalization bounds, that's something different. However, the question was about **estimating** the generalization error.  
  > **username_2**: You don't know if your validation data actually comes from the same distribution or.not basically it's all guess, you never know much of anything until you run the system real time.  
  
  ---
  
  