Answers.Body: <p>Short answer: training "duration" or number of epochs/updates should be cross-validated too: you want to early-stop your training to prevent overfitting.</p>
  
  <p>Longer answer:</p>
  
  <blockquote>
    <p>Think of accuracy on the validation set as an estimate of accuracy on future data, given the value of some hyperparameter. In this case, the hyperparameter of interest is the number of training epochs. So: for each CV fold, train the network (e.g. up to some maximum number of epochs). After each epoch, record accuracy on the validation set. Compute the average validation set accuracy (across CV folds) for each number of epochs. Choose the number of epochs that maximizes this value.</p>
  </blockquote>
  
  <p><a href="https://stats.stackexchange.com/questions/298084/k-fold-cross-validation-for-choosing-number-of-epochs">https://stats.stackexchange.com/questions/298084/k-fold-cross-validation-for-choosing-number-of-epochs</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-09T00:00:14.890Z
Answers.Text: I have some problems with this but I'm having a hard time articulating them. I think at its core given the nature of SGD I don't completely trust this to be correct.
Answers.UserDisplayName: FourierFlux
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-08T23:08:06.003Z
Answers.LastActivityDate: 2020-04-08T23:08:06.003Z
Answers.UserDisplayName: ted
Body: <p>When using k-fold cross-validation in a deep learning problem, after you have computed your hyper-parameters, how do you decide how long to train your final model? My understanding is that, after the hyperparameters are selected, you train your model one more time on the entire set of data, but it's not clear to me when you decide to stop training.  </p>
  
Tags: <machine-learning><deep-learning><training><hyperparameter-optimization><cross-validation>
Title: After having selected the best model with cross-validation, for how long should I train it?
UserDisplayName: FourierFlux
ThreadText: # After having selected the best model with cross-validation, for how long should I train it?
  
  (Asked by: username_0 on 2020-04-08)
  
  When using k-fold cross-validation in a deep learning problem, after you have computed your hyper-parameters, how do you decide how long to train your final model? My understanding is that, after the hyperparameters are selected, you train your model one more time on the entire set of data, but it's not clear to me when you decide to stop training.
  
  **Tags:** machine-learning, deep-learning, training, hyperparameter-optimization, cross-validation
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-04-08. Score: 2
  
  Short answer: training "duration" or number of epochs/updates should be cross-validated too: you want to early-stop your training to prevent overfitting.
  
  Longer answer:
  
  > Think of accuracy on the validation set as an estimate of accuracy on future data, given the value of some hyperparameter. In this case, the hyperparameter of interest is the number of training epochs. So: for each CV fold, train the network (e.g. up to some maximum number of epochs). After each epoch, record accuracy on the validation set. Compute the average validation set accuracy (across CV folds) for each number of epochs. Choose the number of epochs that maximizes this value.
  
  <https://stats.stackexchange.com/questions/298084/k-fold-cross-validation-for-choosing-number-of-epochs>
  
  > **Comments:**
  > **username_0**: I have some problems with this but I'm having a hard time articulating them. I think at its core given the nature of SGD I don't completely trust this to be correct.  
  
  ---
  
  