Answers.Body: <p>Check if the CV and test are Temporaly stable or they change with time.
  Additionally check if they both follow the same distribution or not</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-24T08:44:10.270Z
Answers.LastActivityDate: 2022-11-24T08:44:10.270Z
Answers.UserDisplayName: Banarasi Vaibhav
Answers.Body: <p>When splitting a dataset into a train, validation &amp; test-set you make an important assumption on the data you have. This is called the i.i.d assumption.
  You assume that all observations are:</p>
  <ol>
  <li>sampled independently</li>
  <li>identically distributed</li>
  </ol>
  <p>Let's explore further:</p>
  <p><strong>Independence assumption:</strong><br />
  There should be no known correlations between samples in train and test set. If there are known correlations in the subsets, then account for this! An example of this could be: a model that predicts the job-satisfaction for Twitterusers (so per user a prediction). If a user has tweets present in all subsets, then there is a 'information leakage', so the test-score will be significantly higher (but unreliable).</p>
  <p><strong>Identically Distributed assumption:</strong><br />
  Your train, validation &amp; test set should be identically distributed, so all samples should come from the same distribution. If this is not the case, then your model will not generalise well to your test set. In your case, I believe your train &amp; validation set to be identically distributed, but your test set is not.
  Another example:
  Let's say we train a model that wants to predict how many people take their bike to work on a given weekday (so the only feature of this model is: the day of the week). If our train set only contains data from the summer and our test set only contains data from the winter, then the model will probably not generalise well to the test data. During the winter people would be less likely to take their bike to work than during the summer, so the underlying distributions are not the same. To have a better performance you could for example:</p>
  <ul>
  <li>Add data, which is not always possible (e.g. a dummy variable 'season')</li>
  <li>Change the split</li>
  </ul>
  <p><em>Changing the split</em><br />
  Note that if the actual problem is that your test data does not come from the same distribution as 'real, unseen data' (but your train &amp; validation data do), then doing this might make your model less powerful to generalise to unseen data then it is right now.</p>
  <p>So in your case, it could be a violation of the identically distributed assumption.
  It could be useful to elaborate further on the nature of the data you are using. I believe this might help you to get a clear answer.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-25T18:27:05.790Z
Answers.LastActivityDate: 2022-11-26T08:10:36.780Z
Answers.LastEditDate: 2022-11-26T08:10:36.780Z
Answers.UserDisplayName: HunkBenny
Body: <p>I've trained my artificial neural network, and, as per standard practice, I've picked out the one neural network throughout training that did the best on my validation dataset. That is, the neural network learned from the training data, and generalized to the validation data.</p>
  <p>However, when I run the neural network on the test data, it performs poorly. <strong>What should I do next?</strong></p>
  <p>From my understanding of the theoretical framework, the goal of validation is to ensure that that the network's parameters don't overfit to the training set. (If they do, we'll detect it because the validation score will be bad.) However, the goal of an additional test dataset set beyond the validation dataset is to ensure that our hyperparameters don't overfit. In most scenarios, we train multiple models with different learning rates, etc., and pick the one that does the best on the validation dataset. However, we might just be cherry picking the one that does the best for that validation dataset and doesn't generalize to a test set. So, we add an extra test set to detect if that happens.</p>
  <p>My question is about an analogous case, except for that my hyperparameter is just the training step of the model. I picked out the model that's checkpointed as having the highest validation score. But, when I run it on the test set, it does poorly, showing that I've cherry picked the model with the highest validation score but it still doesn't generalize.</p>
  <p><strong>What do I do next in this scenario?</strong> Do I just follow the same advice from <a href="https://stats.stackexchange.com/questions/365778/what-should-i-do-when-my-neural-network-doesnt-generalize-well">these</a> <a href="https://stats.stackexchange.com/questions/392676/my-neural-net-can-overfit-but-not-generalize">questions</a> about overfitting, or is this a special case because the model does seem to generalize to the validation dataset?</p>
  <p>(Note: this question is different than regular overfitting because it's about hyperparameters overfitting the validation set, not regular parameters overfitting the train set. I've also looked at <a href="https://stats.meta.stackexchange.com/questions/5273/whats-the-best-way-to-answer-my-neural-network-doesnt-work-please-fix-quest">these guidelines</a> but they don't seem to apply to this more general theoretical question.)</p>
  
Tags: <neural-networks><hyperparameter-optimization><overfitting><hyper-parameters><computational-learning-theory>
Title: What should I do if my validation score is good, but my test score is bad?
UserDisplayName: Pro Q
ThreadText: # What should I do if my validation score is good, but my test score is bad?
  
  (Asked by: username_0 on 2022-11-24)
  
  I've trained my artificial neural network, and, as per standard practice, I've picked out the one neural network throughout training that did the best on my validation dataset. That is, the neural network learned from the training data, and generalized to the validation data.
  
  However, when I run the neural network on the test data, it performs poorly. **What should I do next?**
  
  From my understanding of the theoretical framework, the goal of validation is to ensure that that the network's parameters don't overfit to the training set. (If they do, we'll detect it because the validation score will be bad.) However, the goal of an additional test dataset set beyond the validation dataset is to ensure that our hyperparameters don't overfit. In most scenarios, we train multiple models with different learning rates, etc., and pick the one that does the best on the validation dataset. However, we might just be cherry picking the one that does the best for that validation dataset and doesn't generalize to a test set. So, we add an extra test set to detect if that happens.
  
  My question is about an analogous case, except for that my hyperparameter is just the training step of the model. I picked out the model that's checkpointed as having the highest validation score. But, when I run it on the test set, it does poorly, showing that I've cherry picked the model with the highest validation score but it still doesn't generalize.
  
  **What do I do next in this scenario?** Do I just follow the same advice from [these](https://stats.stackexchange.com/questions/365778/what-should-i-do-when-my-neural-network-doesnt-generalize-well) [questions](https://stats.stackexchange.com/questions/392676/my-neural-net-can-overfit-but-not-generalize) about overfitting, or is this a special case because the model does seem to generalize to the validation dataset?
  
  (Note: this question is different than regular overfitting because it's about hyperparameters overfitting the validation set, not regular parameters overfitting the train set. I've also looked at [these guidelines](https://stats.meta.stackexchange.com/questions/5273/whats-the-best-way-to-answer-my-neural-network-doesnt-work-please-fix-quest) but they don't seem to apply to this more general theoretical question.)
  
  **Tags:** neural-networks, hyperparameter-optimization, overfitting, hyper-parameters, computational-learning-theory
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2022-11-25. Score: 2 | Accepted answer
  
  When splitting a dataset into a train, validation & test-set you make an important assumption on the data you have. This is called the i.i.d assumption.
  You assume that all observations are:
  
  1. sampled independently
  2. identically distributed
  
  Let's explore further:
  
  **Independence assumption:**  
  There should be no known correlations between samples in train and test set. If there are known correlations in the subsets, then account for this! An example of this could be: a model that predicts the job-satisfaction for Twitterusers (so per user a prediction). If a user has tweets present in all subsets, then there is a 'information leakage', so the test-score will be significantly higher (but unreliable).
  
  **Identically Distributed assumption:**  
  Your train, validation & test set should be identically distributed, so all samples should come from the same distribution. If this is not the case, then your model will not generalise well to your test set. In your case, I believe your train & validation set to be identically distributed, but your test set is not.
  Another example:
  Let's say we train a model that wants to predict how many people take their bike to work on a given weekday (so the only feature of this model is: the day of the week). If our train set only contains data from the summer and our test set only contains data from the winter, then the model will probably not generalise well to the test data. During the winter people would be less likely to take their bike to work than during the summer, so the underlying distributions are not the same. To have a better performance you could for example:
  
  * Add data, which is not always possible (e.g. a dummy variable 'season')
  * Change the split
  
  *Changing the split*  
  Note that if the actual problem is that your test data does not come from the same distribution as 'real, unseen data' (but your train & validation data do), then doing this might make your model less powerful to generalise to unseen data then it is right now.
  
  So in your case, it could be a violation of the identically distributed assumption.
  It could be useful to elaborate further on the nature of the data you are using. I believe this might help you to get a clear answer.
  
  
  ---
  
  ### Answer by username_1 on 2022-11-24. Score: 1
  
  Check if the CV and test are Temporaly stable or they change with time.
  Additionally check if they both follow the same distribution or not
  
  
  ---
  
  