Answers.Body: <pre><code>Validation dataset: 600 * 24 = 14400
  </code></pre>
  
  <p>Means that you are augmenting the validation set, right? For an experiment, you can do that and it might take validation accuracy more than train accuracy?</p>
  
  <p>The idea of augmentation in only valid for the training set and you should not change the validation set or test set.  </p>
  
  <p>You can try without the augmentation in the validation set and see the result.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-12T09:20:49.757Z
Answers.Text: No, I have not augmented the validation dataset. I have 600 (images/hand sign) x 24 (24 hand signs) = 14400 validation dataset images
Answers.UserDisplayName: mayuresh_sa
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-11T01:15:21.047Z
Answers.LastActivityDate: 2020-03-11T01:15:21.047Z
Answers.UserDisplayName: Ta_Req
Answers.Body: <ol>
  <li>Assuming you pass through the entire validation dataset, this can't be due to shuffling since you still compute the loss/accuracy over the entire dataset, so order does not really matter here. It is more likely that you have a <strong><em>significantly smaller or less representative validation dataset</em></strong>, e.g., distribution of the validation dataset can be skewed towards classes where your model performs better.</li>
  <li>What do you mean exactly by improving the situation? Having a better validation accuracy is not necessarily bad. In any case, if you decrease the effect of regularization, e.g., lowering weight decay, training accuracy might go up but your model might generalize worse, i.e.,  you might get a lower validation accuracy.</li>
  <li>No, the goal of training is never to maximize training accuracy. You can trivially do so by just memorizing the training dataset. In short, the goal of training is to get good generalization and as long as you get a satisfactory validation accuracy, it is likely that this has happened to some degree (assuming you have a <strong><em>good</em></strong> validation dataset of course).</li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-12T09:19:21.737Z
Answers.Text: Thank you for the reply. What does it mean to have good representative validation dataset? Just to understand it, my images were taken on plain and complex backgrounds, so both these types of images are present in the training and validation dataset. Should I segregate them like - Training data with plain bakgrounds and Validation dataset with more complex bakgrounds?
Answers.UserDisplayName: mayuresh_sa
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-12T12:09:07.857Z
Answers.Text: Although I see higher accuracies, it performs significantly poor on several test images I checked. So, it leads to a confusion to how much should I train?
Answers.UserDisplayName: mayuresh_sa
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-13T02:29:17.953Z
Answers.Text: Briefly, the format of the images and the distribution of the classes in your validation set should ideally be the same as your training dataset. You can use the 'early stopping' criteria to decide when to stop training. That is, just keep training as long as your validation loss decreases. Then, use the model snapshotted at lowest validation loss to do inference on test data. Hope that helps.
Answers.UserDisplayName: SpiderRico
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-13T07:46:44.463Z
Answers.Text: Regarding the equal distribution of data for each class, I do not think it is a problem since I have segregated them in different folders. So each class has 2400 training images, 600 validation images. What do you mean by format? The resolutions of all the images are the same. Thanks for the tip to use 'val_loss' as a metric in earlystopping.
Answers.UserDisplayName: mayuresh_sa
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-10T21:41:34.850Z
Answers.LastActivityDate: 2020-03-10T21:41:34.850Z
Answers.UserDisplayName: SpiderRico
Body: <p>I am training a classifier to identify 24 hand signs of American Sign Language. I created a custom dataset by recording videos in different backgrounds for each of the signs and later converted the videos into images. Each sign has 3000 images, that were randomly selected to generate a training dataset with 2400 images/sign and validation dataset with the remaining 600 images/sign.</p>
  
  <ul>
  <li>Total number of images in entire dataset: 3000 * 24 = 72000</li>
  <li>Training dataset: 2400 * 24 = 57600</li>
  <li>Validation dataset: 600 * 24 = 14400</li>
  <li>Image dimension (Width x Height): 1280 x 720 pixels</li>
  </ul>
  
  <h3>The CNN architecture used for training</h3>
  
  <pre><code>model = Sequential([
      Conv2D(32, (3, 3), activation='relu', input_shape=(IMG_HEIGHT, IMG_WIDTH ,3)),
      MaxPooling2D(pool_size=(2,2)),
      Dropout(0.25),
  
      Conv2D(32, (3, 3), activation='relu'),
      MaxPooling2D(pool_size=(2,2)),
      Dropout(0.25),
  
      Conv2D(64, (3, 3), activation='relu'),
      MaxPooling2D(pool_size=(2,2)),
      Dropout(0.25),
  
      Conv2D(64, (3, 3), activation='relu'),
      MaxPooling2D(pool_size=(2,2)),
      Dropout(0.25),
  
      Flatten(),
      Dense(128, activation='relu'),
      Dropout(0.25),
  
      Dense(NUM_CLASSES, activation='softmax')
  ])
  </code></pre>
  
  <h3>Training parameters:</h3>
  
  <pre><code>IMG_HEIGHT = 224
  IMG_WIDTH = 224
  BATCH_SIZE = 32
  NUM_CLASSES = 24
  train_datagen = ImageDataGenerator(rescale = 1./255,
                                     width_shift_range=0.1,
                                     height_shift_range=0.1,
                                     zoom_range=0.1,
                                     fill_mode='constant')
  EPOCHS = 20
  STEPS_PER_EPOCH = TRAIN_TOTAL // BATCH_SIZE
  VALIDATION_STEPS = VALIDATION_TOTAL // BATCH_SIZE
  
  callbacks_list = [
      tf.keras.callbacks.EarlyStopping(monitor = 'accuracy',
                                       min_delta = 0.005,
                                       patience = 3),
      tf.keras.callbacks.ModelCheckpoint(filepath = 'D:\\Models\\HSRS_ThesisDataset_5Mar_1330.h5',
                                         monitor= 'val_loss',
                                         save_best_only = True)
  ]
  
  optimizer = 'adam'
  </code></pre>
  
  <p>The model accuracy and model loss graph is shown in the figure below:</p>
  
  <p><a href="https://i.sstatic.net/tQdab.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/tQdab.png" alt="Model Accuracy and Loss"></a></p>
  
  <h3>The results obtained at the end of the training are</h3>
  
  <ul>
  <li>Train acc: 0.8000121</li>
  <li>Val acc: 0.914441</li>
  </ul>
  
  <p>I read <a href="https://www.pyimagesearch.com/2019/10/14/why-is-my-validation-loss-lower-than-my-training-loss/" rel="nofollow noreferrer">this article explaining why the validation loss is lower than the training loss</a> I want to know:</p>
  
  <ol>
  <li>Is it because of the smaller dataset and random shuffling of the images?</li>
  <li>Is there any way to improve the condition without changing the dataset?</li>
  <li>Will this have a very detrimental effect on the model performance in real test cases? If not, can I just focus on improving the training accuracy of the overall model?</li>
  </ol>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-11T11:37:48.747Z
Comments.Text: That article you're mentioning tells you that dropout can decrease the training performance and you are using dropout. Have you tried not to use dropout during training and see if something changes?
Comments.UserDisplayName: nbro
Tags: <neural-networks><deep-learning><convolutional-neural-networks><computer-vision><datasets>
Title: Why is the validation performance better than the training performance?
UserDisplayName: mayuresh_sa
ThreadText: # Why is the validation performance better than the training performance?
  
  (Asked by: username_0 on 2020-03-10)
  
  I am training a classifier to identify 24 hand signs of American Sign Language. I created a custom dataset by recording videos in different backgrounds for each of the signs and later converted the videos into images. Each sign has 3000 images, that were randomly selected to generate a training dataset with 2400 images/sign and validation dataset with the remaining 600 images/sign.
  
  Total number of images in entire dataset: 3000 * 24 = 72000
  Training dataset: 2400 * 24 = 57600
  Validation dataset: 600 * 24 = 14400
  Image dimension (Width x Height): 1280 x 720 pixels
  
  The CNN architecture used for training
  model = Sequential([
      Conv2D(32, (3, 3), activation='relu', input_shape=(IMG_HEIGHT, IMG_WIDTH ,3)),
      MaxPooling2D(pool_size=(2,2)),
      Dropout(0.25),
  
      Conv2D(32, (3, 3), activation='relu'),
      MaxPooling2D(pool_size=(2,2)),
      Dropout(0.25),
  
      Conv2D(64, (3, 3), activation='relu'),
      MaxPooling2D(pool_size=(2,2)),
      Dropout(0.25),
  
      Conv2D(64, (3, 3), activation='relu'),
      MaxPooling2D(pool_size=(2,2)),
      Dropout(0.25),
  
      Flatten(),
      Dense(128, activation='relu'),
      Dropout(0.25),
  
      Dense(NUM_CLASSES, activation='softmax')
  ])
  
  Training parameters:
  IMG_HEIGHT = 224
  IMG_WIDTH = 224
  BATCH_SIZE = 32
  NUM_CLASSES = 24
  train_datagen = ImageDataGenerator(rescale = 1./255,
                                     width_shift_range=0.1,
                                     height_shift_range=0.1,
                                     zoom_range=0.1,
                                     fill_mode='constant')
  EPOCHS = 20
  STEPS_PER_EPOCH = TRAIN_TOTAL // BATCH_SIZE
  VALIDATION_STEPS = VALIDATION_TOTAL // BATCH_SIZE
  
  callbacks_list = [
      tf.keras.callbacks.EarlyStopping(monitor = 'accuracy',
                                       min_delta = 0.005,
                                       patience = 3),
      tf.keras.callbacks.ModelCheckpoint(filepath = 'D:\\Models\\HSRS_ThesisDataset_5Mar_1330.h5',
                                         monitor= 'val_loss',
                                         save_best_only = True)
  ]
  
  optimizer = 'adam'
  
  The model accuracy and model loss graph is shown in the figure below:
  
  The results obtained at the end of the training are
  
  Train acc: 0.8000121
  Val acc: 0.914441
  
  I read this article explaining why the validation loss is lower than the training loss I want to know:
  
  Is it because of the smaller dataset and random shuffling of the images?
  Is there any way to improve the condition without changing the dataset?
  Will this have a very detrimental effect on the model performance in real test cases? If not, can I just focus on improving the training accuracy of the overall model?
  
  **Tags:** neural-networks, deep-learning, convolutional-neural-networks, computer-vision, datasets
  > **Comments:**
  > **username_1**: That article you're mentioning tells you that dropout can decrease the training performance and you are using dropout. Have you tried not to use dropout during training and see if something changes?  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2020-03-10. Score: 3 | Accepted answer
  
  1. Assuming you pass through the entire validation dataset, this can't be due to shuffling since you still compute the loss/accuracy over the entire dataset, so order does not really matter here. It is more likely that you have a ***significantly smaller or less representative validation dataset***, e.g., distribution of the validation dataset can be skewed towards classes where your model performs better.
  2. What do you mean exactly by improving the situation? Having a better validation accuracy is not necessarily bad. In any case, if you decrease the effect of regularization, e.g., lowering weight decay, training accuracy might go up but your model might generalize worse, i.e., you might get a lower validation accuracy.
  3. No, the goal of training is never to maximize training accuracy. You can trivially do so by just memorizing the training dataset. In short, the goal of training is to get good generalization and as long as you get a satisfactory validation accuracy, it is likely that this has happened to some degree (assuming you have a ***good*** validation dataset of course).
  
  > **Comments:**
  > **username_0**: Thank you for the reply. What does it mean to have good representative validation dataset? Just to understand it, my images were taken on plain and complex backgrounds, so both these types of images are present in the training and validation dataset. Should I segregate them like - Training data with plain bakgrounds and Validation dataset with more complex bakgrounds?  
  > **username_0**: Although I see higher accuracies, it performs significantly poor on several test images I checked. So, it leads to a confusion to how much should I train?  
  > **username_3**: Briefly, the format of the images and the distribution of the classes in your validation set should ideally be the same as your training dataset. You can use the 'early stopping' criteria to decide when to stop training. That is, just keep training as long as your validation loss decreases. Then, use the model snapshotted at lowest validation loss to do inference on test data. Hope that helps.  
  > **username_0**: Regarding the equal distribution of data for each class, I do not think it is a problem since I have segregated them in different folders. So each class has 2400 training images, 600 validation images. What do you mean by format? The resolutions of all the images are the same. Thanks for the tip to use 'val_loss' as a metric in earlystopping.  
  
  ---
  
  ### Answer by username_2 on 2020-03-11. Score: 0
  
  Validation dataset: 600 * 24 = 14400
  
  Means that you are augmenting the validation set, right? For an experiment, you can do that and it might take validation accuracy more than train accuracy?
  The idea of augmentation in only valid for the training set and you should not change the validation set or test set.  
  You can try without the augmentation in the validation set and see the result.
  
  > **Comments:**
  > **username_0**: No, I have not augmented the validation dataset. I have 600 (images/hand sign) x 24 (24 hand signs) = 14400 validation dataset images  
  
  ---
  
  