Answers.Body: <p>First of all, as mentioned by @Neil Slater in the comment - you need to have three splits into the train, validation and test set.</p>
  <p>One sometimes disregards the difference between validation and test set. However they serve for different purposes. Here I would like to cite <a href="https://machinelearningmastery.com/difference-test-validation-datasets/" rel="nofollow noreferrer">https://machinelearningmastery.com/difference-test-validation-datasets/</a> :</p>
  <blockquote>
  <p><strong>Validation Dataset</strong>: The sample of data used to provide an unbiased
  evaluation of a model fit on the training dataset while tuning model
  hyperparameters. The evaluation becomes more biased as skill on the
  validation dataset is incorporated into the model configuration.</p>
  <p><strong>Test Dataset</strong>: The sample of data used to provide an unbiased
  evaluation of a final model fit on the training dataset.</p>
  </blockquote>
  <p>Secondly, in order to understand what's happening plot jointly the train and validation loss. In case the performance on validation data becomes much worse, that on the training - It is better to terminate training, since it is the indication of overfitting.</p>
  <p>A good practice is to use early stopping, there is an implementation of this callback in Tensorflow - <a href="https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/EarlyStopping" rel="nofollow noreferrer">https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/EarlyStopping</a>.</p>
  <p>It a kind of regularization procedure <a href="https://en.wikipedia.org/wiki/Early_stopping" rel="nofollow noreferrer">https://en.wikipedia.org/wiki/Early_stopping</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-06T15:19:19.930Z
Answers.Text: Does anyone talk about "orders of validation".  Ie, you have your order-0 validation (your training set), your order-1 validation (your validation set, which is used to determine what your model hyperparameters should be), then order-2 validation (your "test" dataset, which is not used to tune hyperparameters, but to determine if you screwed up), your order-3 validation (which you use only if your "test" dataset told you that your result was junk); the idea is that order-N validation is used to validate *after previous validations where used as input into your model*, which adds bias.
Answers.UserDisplayName: Yakk
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-06T15:21:37.423Z
Answers.Text: @Yakk Maybe you're talking about **k-fold cross-validation**?
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-06T15:34:39.183Z
Answers.Text: @nbro No, something different. Yakk is generalizing the difference between test and validation sets. The reason you need two is because your training loop has two levels of nesting. If you somehow needed, say, three more levels of nesting, you'd need 5 of those sets.
Answers.UserDisplayName: Criticize SE actions means ban
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-06T15:39:05.260Z
Answers.Text: As far as I am aware, this idea is not common in ML, but it's possible that something along those lines has been applied. Typically, you use the same validation data for early stopping **or/and** hyper-parameter tuning. I agree that, if you used a certain validation dataset for hyper-parameter tuning, then selected a specific function using another validation dataset, you could be biased (and the bias would come from the choice of the specific hyper-parameters with the previous val. data, which, of course, also limits the specific function from the specific hypothesis class you can choose).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-06T08:43:46.810Z
Answers.LastActivityDate: 2021-07-06T14:23:22.030Z
Answers.LastEditDate: 2021-07-06T14:23:22.030Z
Answers.UserDisplayName: spiridon_the_sun_rotator
Body: <p>I have a neural network where there are two hidden layers. Each hidden layer has 128 neurons. The input layer has 20 inputs, and the output layer has 3 outputs.</p>
  <p>I have 1 million records of data. 80% is used to train the network, 20% is used for validation. I run the training for 100000 epochs.</p>
  <p>I see that the neural network attains 100% accuracy on the training data after only 12000 epochs.</p>
  <p>Should I stop training or continue until all 100000 epochs are complete? Please, explain why.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-06T06:20:37.660Z
Comments.Text: You don't appear to have enough splits for normal best practice here. There should be a three-way split. How are you selecting between models, and following that how are you assessing accuracy in an unbiased fashion?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-06T11:50:44.900Z
Comments.Text: As a rule of thumb, generally 100% training accuracy means you've overtrained. Training accuracy isn't what you should be looking at anyway, as mentioned in the other comments, you should be looking at validation accuracy.
Comments.UserDisplayName: Recessive
Tags: <neural-networks><training><overfitting><accuracy><cross-validation>
Title: Should I continue training if the neural network attains 100% training accuracy?
UserDisplayName: user366312
ThreadText: # Should I continue training if the neural network attains 100% training accuracy?
  
  (Asked by: username_0 on 2021-07-06)
  
  I have a neural network where there are two hidden layers. Each hidden layer has 128 neurons. The input layer has 20 inputs, and the output layer has 3 outputs.
  
  I have 1 million records of data. 80% is used to train the network, 20% is used for validation. I run the training for 100000 epochs.
  
  I see that the neural network attains 100% accuracy on the training data after only 12000 epochs.
  
  Should I stop training or continue until all 100000 epochs are complete? Please, explain why.
  
  **Tags:** neural-networks, training, overfitting, accuracy, cross-validation
  
  > **Comments:**
  > **username_1**: You don't appear to have enough splits for normal best practice here. There should be a three-way split. How are you selecting between models, and following that how are you assessing accuracy in an unbiased fashion?  
  > **username_2**: As a rule of thumb, generally 100% training accuracy means you've overtrained. Training accuracy isn't what you should be looking at anyway, as mentioned in the other comments, you should be looking at validation accuracy.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2021-07-06. Score: 6
  
  First of all, as mentioned by @username_1 in the comment - you need to have three splits into the train, validation and test set.
  
  One sometimes disregards the difference between validation and test set. However they serve for different purposes. Here I would like to cite <https://machinelearningmastery.com/difference-test-validation-datasets/> :
  
  > **Validation Dataset**: The sample of data used to provide an unbiased
  > evaluation of a model fit on the training dataset while tuning model
  > hyperparameters. The evaluation becomes more biased as skill on the
  > validation dataset is incorporated into the model configuration.
  >
  > **Test Dataset**: The sample of data used to provide an unbiased
  > evaluation of a final model fit on the training dataset.
  
  Secondly, in order to understand what's happening plot jointly the train and validation loss. In case the performance on validation data becomes much worse, that on the training - It is better to terminate training, since it is the indication of overfitting.
  
  A good practice is to use early stopping, there is an implementation of this callback in Tensorflow - <https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/EarlyStopping>.
  
  It a kind of regularization procedure <https://en.wikipedia.org/wiki/Early_stopping>.
  
  > **Comments:**
  > **username_4**: Does anyone talk about "orders of validation".  Ie, you have your order-0 validation (your training set), your order-1 validation (your validation set, which is used to determine what your model hyperparameters should be), then order-2 validation (your "test" dataset, which is not used to tune hyperparameters, but to determine if you screwed up), your order-3 validation (which you use only if your "test" dataset told you that your result was junk); the idea is that order-N validation is used to validate *after previous validations where used as input into your model*, which adds bias.  
  > **username_5**: @username_4 Maybe you're talking about **k-fold cross-validation**?  
  > **username_6**: @username_5 No, something different. username_4 is generalizing the difference between test and validation sets. The reason you need two is because your training loop has two levels of nesting. If you somehow needed, say, three more levels of nesting, you'd need 5 of those sets.  
  > **username_5**: As far as I am aware, this idea is not common in ML, but it's possible that something along those lines has been applied. Typically, you use the same validation data for early stopping **or/and** hyper-parameter tuning. I agree that, if you used a certain validation dataset for hyper-parameter tuning, then selected a specific function using another validation dataset, you could be biased (and the bias would come from the choice of the specific hyper-parameters with the previous val. data, which, of course, also limits the specific function from the specific hypothesis class you can choose).  
  
  ---
  
  