Answers.Body: <p>The purpose of the test set is to test your model before deploying, otherwise, you would not need the test set in the first place. If you retrain your model by also including the validation and test datasets, of course, you cannot test your model anymore. You need to leave the test dataset separate and not use it for retraining, unless you have more data for testing.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-25T13:22:13.323Z
Answers.Text: But the final model, deployed on production should be trained on all data. I just wonder how to be sure that such model will work well in future.
Answers.UserDisplayName: GKozinski
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-25T13:25:06.667Z
Answers.Text: @Makintosz As I say in my answer, I don't know where you read this, but, no, the final model should not be trained on all data (as far as I am concerned), unless you are willing not to be able to test the model you will deploy. There's no way to test your model without data. In the end, your model is a machine learning model that learns from and is evaluated on data.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-25T13:55:31.613Z
Answers.Text: First source I could find: https://machinelearningmastery.com/train-final-machine-learning-model/ it says: "Your model will likely perform better when trained on all of the available data than just the subset used to estimate the performance of the model.
  This is why we prefer to train the final model on all available data."
Answers.UserDisplayName: GKozinski
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-25T14:02:28.923Z
Answers.Text: @Makintosz Well, it's true that the model will likely perform better given more data, but then you cannot empirically evaluate it anymore if you don't have more data. However, I've just seen that someone suggested looking into stuff like PAC learning and error bounds. Maybe that serves your purpose.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-24T23:56:53.697Z
Answers.LastActivityDate: 2020-08-24T23:56:53.697Z
Answers.UserDisplayName: nbro
Answers.Body: <p>We usually divide the dataset into multiple subsets namely (training, validation and test sets). During training, we validate the model against the validation set. And during testing, we use the test dataset to obtain metrics for the model. We should make sure the subsets are taken from the same sample. Once you've tested it against the test subset, there's nothing really we can do.</p>
  <p>You can also increase your dataset, by using multiple data sources if the problem statement allows you to.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-24T03:35:43.680Z
Answers.LastActivityDate: 2020-09-24T03:35:43.680Z
Answers.UserDisplayName: Saurav Maheshkar
Body: <p>The 'by the book' method of delivering final machine learning models is to include all data in the final training (including validation and test sets). To check robustness of my model I use randomly chosen population for training and validation sets with each training (no set random seed). The results on validation and then test sets are pretty satisfactory for my case however they are always different each time, precision spans between 0.7 and 0.9. This is due to fact that each time different data points fall to set with which model is trained.</p>
  <p>My question is: how do I know that final training will also generate good model and how to estimate its precision when I do not have anymore unseen data?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-25T00:07:21.117Z
Comments.Text: I'm not sure if I understand your question correctly, but PAC model (probably approximately correct) offers some probabilistic performance guarantees on the trained models from the training accuracy and so on. Maybe this is what you're looking for: https://en.wikipedia.org/wiki/Probably_approximately_correct_learning
Comments.UserDisplayName: SpiderRico
Tags: <machine-learning><training><models><testing>
Title: How can I be sure that the final model, trained on all data, is correct?
UserDisplayName: GKozinski
ThreadText: # How can I be sure that the final model, trained on all data, is correct?
  
  (Asked by: username_0 on 2020-08-23)
  
  The 'by the book' method of delivering final machine learning models is to include all data in the final training (including validation and test sets). To check robustness of my model I use randomly chosen population for training and validation sets with each training (no set random seed). The results on validation and then test sets are pretty satisfactory for my case however they are always different each time, precision spans between 0.7 and 0.9. This is due to fact that each time different data points fall to set with which model is trained.
  
  My question is: how do I know that final training will also generate good model and how to estimate its precision when I do not have anymore unseen data?
  
  **Tags:** machine-learning, training, models, testing
  
  > **Comments:**
  > **username_1**: I'm not sure if I understand your question correctly, but PAC model (probably approximately correct) offers some probabilistic performance guarantees on the trained models from the training accuracy and so on. Maybe this is what you're looking for: https://en.wikipedia.org/wiki/Probably_approximately_correct_learning  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2020-08-24. Score: 2
  
  The purpose of the test set is to test your model before deploying, otherwise, you would not need the test set in the first place. If you retrain your model by also including the validation and test datasets, of course, you cannot test your model anymore. You need to leave the test dataset separate and not use it for retraining, unless you have more data for testing.
  
  > **Comments:**
  > **username_0**: But the final model, deployed on production should be trained on all data. I just wonder how to be sure that such model will work well in future.  
  > **username_2**: @Makintosz As I say in my answer, I don't know where you read this, but, no, the final model should not be trained on all data (as far as I am concerned), unless you are willing not to be able to test the model you will deploy. There's no way to test your model without data. In the end, your model is a machine learning model that learns from and is evaluated on data.  
  > **username_0**: First source I could find: https://machinelearningmastery.com/train-final-machine-learning-model/ it says: "Your model will likely perform better when trained on all of the available data than just the subset used to estimate the performance of the model. This is why we prefer to train the final model on all available data."  
  > **username_2**: @Makintosz Well, it's true that the model will likely perform better given more data, but then you cannot empirically evaluate it anymore if you don't have more data. However, I've just seen that someone suggested looking into stuff like PAC learning and error bounds. Maybe that serves your purpose.  
  
  ---
  
  ### Answer by username_3 on 2020-09-24. Score: 1
  
  We usually divide the dataset into multiple subsets namely (training, validation and test sets). During training, we validate the model against the validation set. And during testing, we use the test dataset to obtain metrics for the model. We should make sure the subsets are taken from the same sample. Once you've tested it against the test subset, there's nothing really we can do.
  
  You can also increase your dataset, by using multiple data sources if the problem statement allows you to.
  
  
  ---
  
  