Answers.Body: <p>Running for "to many" epochs can indeed lead to over fitting. You should look at the validation loss. If on AVERAGE it continues to decrease then you are not yet over fitting.
  You may be tempted to run more epochs in hopes your loss will decrease but unless you adjust your learning rate dynamically  at some point you won't get any improvement.
  If you use KERAS it has a useful callback ReduceLROnPlateau. Documentation is at <a href="https://keras.io/callbacks/" rel="nofollow noreferrer">https://keras.io/callbacks/</a>
  This allows you to monitor a metric (typically validation loss) and to adjust the learning rate by a user defined factor(parameter factor) if the metric you are monitoring fails to improve after a certain number of consecutive epochs(parameter patience). You can think of the training process as travelling down a valley in N space(N being the number of trainable parameters). As you descend towards a minimum they valley gets narrower. If you do not lower the learning rate you will reach a point where you can not descend any further.Now you could use a very small learning rate to begin with but then you will have to train for a lot more epochs. One problem with adjusting the learning rate just on the validation loss is that in the early training epochs validation loss often does not track with training accuracy and it could cause the learning rate to be decreased prematurely.
  I wrote a custom callback which initially monitors training loss and adjust the learning rate based on that metric. Once the training accuracy reaches 95% it switches to monitoring validation loss and adjusts the learning rate based on that.  It saves the model weights for the lowest validation loss in the variable val.best_weights. After training load these weights into your model to make predictions. Code is below if you are interested. When you compile your model just add 'val' to the callback list.</p>
  
  <pre><code> class val(tf.keras.callbacks.Callback):
          # functions in this class adjust the learning rate 
          lowest_loss=np.inf
          lowest_trloss=np.inf
          best_weights=model.get_weights()
          lr=float(tf.keras.backend.get_value(model.optimizer.lr))
          epoch=0
          highest_acc=0
  
          def __init__(self):
              super(val, self).__init__()
              self.lowest_loss=np.inf
              self.lowest_trloss=np.inf
              self.best_weights=model.get_weights()
              self.lr=float(tf.keras.backend.get_value(model.optimizer.lr))
              self.epoch=0
              self.highest_acc=0
  
  
          def on_epoch_end(self, epoch, logs=None):             
              val.lr=float(tf.keras.backend.get_value(self.model.optimizer.lr))
              val.epoch=val.epoch +1            
              v_loss=logs.get('val_loss')
              v_acc=logs.get('accuracy')
              loss=logs.get('loss')
              if loss&lt;val.lowest_trloss:
                  val.lowest_trloss=loss
                  if v_acc&lt;.90:
                      val.best_weights=model.get_weights()
              if v_acc&lt;=.95 and loss&gt;val.lowest_trloss:
                  lr=float(tf.keras.backend.get_value(self.model.optimizer.lr))
                  ratio=val.lowest_trloss/loss  # add a factor to lr reduction
                  new_lr=lr * .7 * ratio
                  tf.keras.backend.set_value(model.optimizer.lr, new_lr)
                  msg='{0}\n current training loss {1:7.5f}  is above lowest training loss of {2:7.5f}, reducing lr to {3:11.9f}{4}'
                  print(msg.format(Cyellow, loss, val.lowest_trloss, new_lr,Cend))   
              if val.lowest_loss &gt; v_loss:
                  msg='{0}\n validation loss improved,saving weights with validation loss= {1:7.4f}\n{2}'
                  print(msg.format(Cgreen, v_loss, Cend))
                  val.lowest_loss=v_loss
                  val.best_weights=model.get_weights()
  
              else:
                   if v_acc&gt;.95 and val.lowest_loss&lt;v_loss:
                          # reduce learning rate based on validation loss&gt; val.best_loss
                          lr=float(tf.keras.backend.get_value(self.model.optimizer.lr))
                          ratio=val.lowest_loss/v_loss  # add a factor to lr reduction
                          new_lr=lr * .7 * ratio
                          tf.keras.backend.set_value(model.optimizer.lr, new_lr)
                          msg='{0}\n current loss {1:7.4f} exceeds lowest loss of {2:7.4f}, reducing lr to {3:11.9f}{4}'
                          print(msg.format(Cyellow, v_loss, val.lowest_loss, new_lr,Cend))```
  
  
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-05T17:15:55.270Z
Answers.LastActivityDate: 2020-03-06T06:02:35.797Z
Answers.LastEditDate: 2020-03-06T06:02:35.797Z
Answers.UserDisplayName: Gerry P
Answers.Body: <p>Training a neural network for "too many" epochs than needed without using early stopping criterion leads to overfitting, where your model's ability to generalize decreases.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-05T13:27:11.133Z
Answers.LastActivityDate: 2020-03-05T13:27:11.133Z
Answers.UserDisplayName: Arun
Body: <p>I have a question about training a neural network for more epochs even after the network has converged without using early stopping criterion.</p>
  
  <p>Consider the MNIST dataset and a LeNet 300-100-10 dense fully-connected architecture, where I have 2 hidden layers having 300 and 100 neurons and an output layer having 10 neurons.</p>
  
  <p>Now, usually, this network takes about 9-11 epochs to train and have a validation accuracy of around 98%.</p>
  
  <p>What happens if I train this network for 25 or 30 epochs, without using early stopping criterion? </p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-03-05T12:18:05.193Z
Comments.Text: Is something preventing you from doing it and observing the results?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-03-05T12:19:17.237Z
Comments.Text: For LeNet 300-100-10 network, no. But for VGG-19, yes, since my GPU isn't powerful enough, and hence my question
Comments.UserDisplayName: Arun
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-03-05T12:24:53.503Z
Comments.Text: Why are you asking about the LeNet network on MNIST then? Have you tried with the LeNet network on MNIST? If so, what happened? Are you asking whether you would get the same effect with VGG-19 on ImageNet?
Comments.UserDisplayName: Neil Slater
Tags: <neural-networks><machine-learning><training><feedforward-neural-networks><early-stopping>
Title: What happens if I train a network for more epochs, without using early stopping?
UserDisplayName: Arun
ThreadText: # What happens if I train a network for more epochs, without using early stopping?
  
  (Asked by: username_0 on 2020-03-05)
  
  I have a question about training a neural network for more epochs even after the network has converged without using early stopping criterion.
  
  Consider the MNIST dataset and a LeNet 300-100-10 dense fully-connected architecture, where I have 2 hidden layers having 300 and 100 neurons and an output layer having 10 neurons.
  
  Now, usually, this network takes about 9-11 epochs to train and have a validation accuracy of around 98%.
  
  What happens if I train this network for 25 or 30 epochs, without using early stopping criterion?
  
  **Tags:** neural-networks, machine-learning, training, feedforward-neural-networks, early-stopping
  > **Comments:**
  > **username_1**: Is something preventing you from doing it and observing the results?  
  > **username_0**: For LeNet 300-100-10 network, no. But for VGG-19, yes, since my GPU isn't powerful enough, and hence my question  
  > **username_1**: Why are you asking about the LeNet network on MNIST then? Have you tried with the LeNet network on MNIST? If so, what happened? Are you asking whether you would get the same effect with VGG-19 on ImageNet?  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2020-03-05. Score: 1 | Accepted answer
  
  Running for "to many" epochs can indeed lead to over fitting. You should look at the validation loss. If on AVERAGE it continues to decrease then you are not yet over fitting.
  You may be tempted to run more epochs in hopes your loss will decrease but unless you adjust your learning rate dynamically  at some point you won't get any improvement.
  If you use KERAS it has a useful callback ReduceLROnPlateau. Documentation is at https://keras.io/callbacks/
  This allows you to monitor a metric (typically validation loss) and to adjust the learning rate by a user defined factor(parameter factor) if the metric you are monitoring fails to improve after a certain number of consecutive epochs(parameter patience). You can think of the training process as travelling down a valley in N space(N being the number of trainable parameters). As you descend towards a minimum they valley gets narrower. If you do not lower the learning rate you will reach a point where you can not descend any further.Now you could use a very small learning rate to begin with but then you will have to train for a lot more epochs. One problem with adjusting the learning rate just on the validation loss is that in the early training epochs validation loss often does not track with training accuracy and it could cause the learning rate to be decreased prematurely.
  I wrote a custom callback which initially monitors training loss and adjust the learning rate based on that metric. Once the training accuracy reaches 95% it switches to monitoring validation loss and adjusts the learning rate based on that.  It saves the model weights for the lowest validation loss in the variable val.best_weights. After training load these weights into your model to make predictions. Code is below if you are interested. When you compile your model just add 'val' to the callback list.
   class val(tf.keras.callbacks.Callback):
          # functions in this class adjust the learning rate 
          lowest_loss=np.inf
          lowest_trloss=np.inf
          best_weights=model.get_weights()
          lr=float(tf.keras.backend.get_value(model.optimizer.lr))
          epoch=0
          highest_acc=0
  
          def __init__(self):
              super(val, self).__init__()
              self.lowest_loss=np.inf
              self.lowest_trloss=np.inf
              self.best_weights=model.get_weights()
              self.lr=float(tf.keras.backend.get_value(model.optimizer.lr))
              self.epoch=0
              self.highest_acc=0
  
  
          def on_epoch_end(self, epoch, logs=None):             
              val.lr=float(tf.keras.backend.get_value(self.model.optimizer.lr))
              val.epoch=val.epoch +1            
              v_loss=logs.get('val_loss')
              v_acc=logs.get('accuracy')
              loss=logs.get('loss')
              if lossval.lowest_trloss:
                  lr=float(tf.keras.backend.get_value(self.model.optimizer.lr))
                  ratio=val.lowest_trloss/loss  # add a factor to lr reduction
                  new_lr=lr * .7 * ratio
                  tf.keras.backend.set_value(model.optimizer.lr, new_lr)
                  msg='{0}\n current training loss {1:7.5f}  is above lowest training loss of {2:7.5f}, reducing lr to {3:11.9f}{4}'
                  print(msg.format(Cyellow, loss, val.lowest_trloss, new_lr,Cend))   
              if val.lowest_loss > v_loss:
                  msg='{0}\n validation loss improved,saving weights with validation loss= {1:7.4f}\n{2}'
                  print(msg.format(Cgreen, v_loss, Cend))
                  val.lowest_loss=v_loss
                  val.best_weights=model.get_weights()
  
              else:
                   if v_acc>.95 and val.lowest_loss val.best_loss
                          lr=float(tf.keras.backend.get_value(self.model.optimizer.lr))
                          ratio=val.lowest_loss/v_loss  # add a factor to lr reduction
                          new_lr=lr * .7 * ratio
                          tf.keras.backend.set_value(model.optimizer.lr, new_lr)
                          msg='{0}\n current loss {1:7.4f} exceeds lowest loss of {2:7.4f}, reducing lr to {3:11.9f}{4}'
                          print(msg.format(Cyellow, v_loss, val.lowest_loss, new_lr,Cend))```
  
  
  ---
  
  ### Answer by username_0 on 2020-03-05. Score: 2
  
  Training a neural network for "too many" epochs than needed without using early stopping criterion leads to overfitting, where your model's ability to generalize decreases.
  
  
  ---
  
  