Answers.Body: <p>The training accuracy tells you nothing about how good it is on other data than the ones it learned on, it could be better on this data because it memorized this examples.</p>
  
  <p>On the other hand the validation set is here to indicate you how good the model is to generalize what it learned to new data (hopefully the testing dataset accurately represents the diversity of the data).</p>
  
  <p>As you are looking for a model which is good on every dataset you don't want to use training accuracy to choose your model and so you should choose the first one.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-05T21:31:41.247Z
Answers.LastActivityDate: 2020-02-05T21:31:41.247Z
Answers.UserDisplayName: kirua
Answers.Body: <p>Neither of the above mentioned methods could be a potent indicator of the performance of a model. </p>
  
  <p>A simple way to train the model <em>just enough</em> so that it generalizes well on unknown datasets would be to monitor the validation loss. Training should be stopped once the validation loss progressively starts increasing over multiple epochs. Beyond this point, the model learns the statistical noise within the data and starts <em>overfitting</em>.</p>
  
  <p><a href="https://i.sstatic.net/EwzMW.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/EwzMW.png" alt="enter image description here"></a></p>
  
  <p>This technique of <strong>Early stopping</strong> could be implemented in Keras with the help of a callback function:</p>
  
  <pre><code>class EarlyStop(tf.keras.callbacks.Callback):
      def on_epoch_end(self, epoch, logs={}):
          if(logs.get('val_loss') &lt; LOSS_THRESHOLD and logs.get('val_categorical_accuracy') &gt; ACCURACY_THRESHOLD):
              self.model.stop_training = True
  
  callbacks= EarlyStop()
  model.fit(...,callbacks=[callbacks])
  </code></pre>
  
  <p>The Loss and Accuracy thresholds can be estimated after a trial run of the model by monitoring the validation/training error graph.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-08T01:28:36.307Z
Answers.Text: So I should track (and log) the `val_loss` metric instead of `val_acc` in the name of output model. Is that right? And choose the lowest one.
Answers.UserDisplayName: Mary
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-08T01:52:07.047Z
Answers.Text: @TinaJ Yes, that would be better indicator of the model performance. Validation accuracy may fluctuate throughout the training procedure (a high validation accuracy reached in the initial epochs could be just a fluke, signifying little about the predictive power of the model).
Answers.UserDisplayName: s_bh
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-08T04:21:05.547Z
Answers.Text: umm I re-train my model once in a while with added data. I see `val_acc` increases but when I test the model expecting better accuracy, it actually behaves poorlier than before. Is it because the `val_loss` was probably higher than before?
Answers.UserDisplayName: Mary
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-08T01:08:24.907Z
Answers.LastActivityDate: 2020-02-08T01:14:05.177Z
Answers.LastEditDate: 2020-02-08T01:14:05.177Z
Answers.UserDisplayName: s_bh
Body: <p>I'm training a deep network in <code>Keras</code> on some images for a binary classification (I have around 12K images). Once in a while, I collect some false positives and add them to my training sets and re-train for higher accuracy.</p>
  <p>I split my training into 20/80 percent for training/validation sets.</p>
  <p>Now, my question is: which resulting model should I use? Always the one with higher validation accuracy, or maybe the higher mean of training and validation accuracy? Which one of the two would you prefer?</p>
  <pre><code>Epoch #38: training acc: 0.924, validation acc: 0.944
  Epoch #90: training acc: 0.952, validation acc: 0.932
  </code></pre>
  
Tags: <deep-learning><training><accuracy><cross-validation><early-stopping>
Title: Should I choose the model with highest validation accuracy or the model with highest mean of training and validation accuracy?
UserDisplayName: Mary
ThreadText: # Should I choose the model with highest validation accuracy or the model with highest mean of training and validation accuracy?
  
  (Asked by: username_0 on 2020-02-05)
  
  I'm training a deep network in `Keras` on some images for a binary classification (I have around 12K images). Once in a while, I collect some false positives and add them to my training sets and re-train for higher accuracy.
  
  I split my training into 20/80 percent for training/validation sets.
  
  Now, my question is: which resulting model should I use? Always the one with higher validation accuracy, or maybe the higher mean of training and validation accuracy? Which one of the two would you prefer?
  
  ```
  Epoch #38: training acc: 0.924, validation acc: 0.944
  Epoch #90: training acc: 0.952, validation acc: 0.932
  
  ```
  
  **Tags:** deep-learning, training, accuracy, cross-validation, early-stopping
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2020-02-08. Score: 3 | Accepted answer
  
  Neither of the above mentioned methods could be a potent indicator of the performance of a model.
  
  A simple way to train the model *just enough* so that it generalizes well on unknown datasets would be to monitor the validation loss. Training should be stopped once the validation loss progressively starts increasing over multiple epochs. Beyond this point, the model learns the statistical noise within the data and starts *overfitting*.
  
  [![enter image description here](https://i.sstatic.net/EwzMW.png)](https://i.sstatic.net/EwzMW.png)
  
  This technique of **Early stopping** could be implemented in Keras with the help of a callback function:
  
  ```
  class EarlyStop(tf.keras.callbacks.Callback):
      def on_epoch_end(self, epoch, logs={}):
          if(logs.get('val_loss') < LOSS_THRESHOLD and logs.get('val_categorical_accuracy') > ACCURACY_THRESHOLD):
              self.model.stop_training = True
  
  callbacks= EarlyStop()
  model.fit(...,callbacks=[callbacks])
  
  ```
  
  The Loss and Accuracy thresholds can be estimated after a trial run of the model by monitoring the validation/training error graph.
  
  > **Comments:**
  > **username_0**: So I should track (and log) the `val_loss` metric instead of `val_acc` in the name of output model. Is that right? And choose the lowest one.  
  > **username_2**: @TinaJ Yes, that would be better indicator of the model performance. Validation accuracy may fluctuate throughout the training procedure (a high validation accuracy reached in the initial epochs could be just a fluke, signifying little about the predictive power of the model).  
  > **username_0**: umm I re-train my model once in a while with added data. I see `val_acc` increases but when I test the model expecting better accuracy, it actually behaves poorlier than before. Is it because the `val_loss` was probably higher than before?  
  
  ---
  
  ### Answer by username_1 on 2020-02-05. Score: 1
  
  The training accuracy tells you nothing about how good it is on other data than the ones it learned on, it could be better on this data because it memorized this examples.
  
  On the other hand the validation set is here to indicate you how good the model is to generalize what it learned to new data (hopefully the testing dataset accurately represents the diversity of the data).
  
  As you are looking for a model which is good on every dataset you don't want to use training accuracy to choose your model and so you should choose the first one.
  
  
  ---
  
  