Answers.Body: <p>I am answering my own question here. The only additional thing that I found was that the average accuracy across a batch of data was slightly higher if the amount of samples in the batch was lower. So the validation set was only 15% of the data, therefore the average accuracy was slightly lower than for 70% of the data. I don't know why the more samples you take the lower the average accuracy, and whether this was a bug in the accuracy calculation or it is the expected behavior. Either way, if you have the same problem, one suggestion is plotting average accuracy vs number of samples and see if this is the reason why you get a lower training accuracy.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-31T01:48:55.187Z
Answers.Text: It's probably that smaller sample sizes are more prone to bias. It could just be the train/val split split the validation data into a bias set. Maybe changing the split could give more expected results?
Answers.UserDisplayName: Recessive
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-02T21:50:31.367Z
Answers.LastActivityDate: 2021-09-02T21:50:31.367Z
Answers.UserDisplayName: Lis Louise
Body: <p>I implemented the unet in TensorFlow for the segmentation of MRI images of the thigh. I noticed I always get a higher validation accuracy  by a small gap, independently of the initial split. One example:</p>
  <p><a href="https://i.sstatic.net/dskC9.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/dskC9.png" alt="enter image description here" /></a></p>
  <p>So I researched when this could be possible:</p>
  <ol>
  <li>When we have an &quot;easy&quot; validation set. I trained it for different initial splitting, all of them showed a higher validation accuracy.</li>
  <li>Regularization and augmentation may reduce the training accuracy. I removed the augmentation and  dropout regularization and still observed the same gap, the only difference was that it took more epochs to reach convergence.</li>
  <li>The last thing I found was that in Keras the training accuracy and loss are averaged over each iteration of the corresponding epoch, while the validation accuracy and loss is calculated from the model at the end of the epoch, which might make the the training loss higher and accuracy lower.</li>
  </ol>
  <p>So I thought that if I train and validate on the same set, then I should get the same curve but shifted by one epoch. So I trained only on 2 batches and validated on the same 2 batches (without dropout or augmentation). I still think there is something else happening because they don't look quite the same and at least at the end when the weights are not changing anymore, the training and validation accuracy should be the same (but still the validation accuracy is higher by a small gap). This is the plot:</p>
  <p><a href="https://i.sstatic.net/9sJlK.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/9sJlK.png" alt="enter image description here" /></a>
  <a href="https://i.sstatic.net/xBUt9.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/xBUt9.png" alt="enter image description here" /></a></p>
  <p>Is there anything else that can be increasing the loss values, this is the model I am using:</p>
  <pre><code>def unet_no_dropout(pretrained_weights=None, input_size=(512, 512, 1)):
  inputs = tf.keras.layers.Input(input_size)
  conv1 = tf.keras.layers.Conv2D(64, 3, activation='relu', padding='same', kernel_initializer='he_normal')(inputs)
  conv1 = tf.keras.layers.Conv2D(64, 3, activation='relu', padding='same', kernel_initializer='he_normal')(conv1)
  pool1 = tf.keras.layers.MaxPooling2D(pool_size=(2, 2))(conv1)
  conv2 = tf.keras.layers.Conv2D(128, 3, activation='relu', padding='same', kernel_initializer='he_normal')(pool1)
  conv2 = tf.keras.layers.Conv2D(128, 3, activation='relu', padding='same', kernel_initializer='he_normal')(conv2)
  pool2 = tf.keras.layers.MaxPooling2D(pool_size=(2, 2))(conv2)
  conv3 = tf.keras.layers.Conv2D(256, 3, activation='relu', padding='same', kernel_initializer='he_normal')(pool2)
  conv3 = tf.keras.layers.Conv2D(256, 3, activation='relu', padding='same', kernel_initializer='he_normal')(conv3)
  pool3 = tf.keras.layers.MaxPooling2D(pool_size=(2, 2))(conv3)
  conv4 = tf.keras.layers.Conv2D(512, 3, activation='relu', padding='same', kernel_initializer='he_normal')(pool3)
  conv4 = tf.keras.layers.Conv2D(512, 3, activation='relu', padding='same', kernel_initializer='he_normal')(conv4)
  #drop4 = tf.keras.layers.Dropout(0.5)(conv4)
  pool4 = tf.keras.layers.MaxPooling2D(pool_size=(2, 2))(conv4)
  
  conv5 = tf.keras.layers.Conv2D(1024, 3, activation='relu', padding='same', kernel_initializer='he_normal')(pool4)
  conv5 = tf.keras.layers.Conv2D(1024, 3, activation='relu', padding='same', kernel_initializer='he_normal')(conv5)
  #drop5 = tf.keras.layers.Dropout(0.5)(conv5)
  
  up6 = tf.keras.layers.Conv2D(512, 2, activation='relu', padding='same', kernel_initializer='he_normal')(
      tf.keras.layers.UpSampling2D(size=(2, 2))(conv5))
  merge6 = tf.keras.layers.concatenate([conv4, up6], axis=3)
  #merge6 = tf.keras.layers.concatenate([conv4, up6], axis=3)
  conv6 = tf.keras.layers.Conv2D(512, 3, activation='relu', padding='same', kernel_initializer='he_normal')(merge6)
  conv6 = tf.keras.layers.Conv2D(512, 3, activation='relu', padding='same', kernel_initializer='he_normal')(conv6)
  
  up7 = tf.keras.layers.Conv2D(256, 2, activation='relu', padding='same', kernel_initializer='he_normal')(
      tf.keras.layers.UpSampling2D(size=(2, 2))(conv6))
  merge7 = tf.keras.layers.concatenate([conv3, up7], axis=3)
  conv7 = tf.keras.layers.Conv2D(256, 3, activation='relu', padding='same', kernel_initializer='he_normal')(merge7)
  conv7 = tf.keras.layers.Conv2D(256, 3, activation='relu', padding='same', kernel_initializer='he_normal')(conv7)
  
  up8 = tf.keras.layers.Conv2D(128, 2, activation='relu', padding='same', kernel_initializer='he_normal')(
      tf.keras.layers.UpSampling2D(size=(2, 2))(conv7))
  merge8 = tf.keras.layers.concatenate([conv2, up8], axis=3)
  conv8 = tf.keras.layers.Conv2D(128, 3, activation='relu', padding='same', kernel_initializer='he_normal')(merge8)
  conv8 = tf.keras.layers.Conv2D(128, 3, activation='relu', padding='same', kernel_initializer='he_normal')(conv8)
  
  up9 = tf.keras.layers.Conv2D(64, 2, activation='relu', padding='same', kernel_initializer='he_normal')(
      tf.keras.layers.UpSampling2D(size=(2, 2))(conv8))
  merge9 = tf.keras.layers.concatenate([conv1, up9], axis=3)
  conv9 = tf.keras.layers.Conv2D(64, 3, activation='relu', padding='same', kernel_initializer='he_normal')(merge9)
  conv9 = tf.keras.layers.Conv2D(64, 3, activation='relu', padding='same', kernel_initializer='he_normal')(conv9)
  conv9 = tf.keras.layers.Conv2D(2, 3, activation='relu', padding='same', kernel_initializer='he_normal')(conv9)
  conv10 = tf.keras.layers.Conv2D(1, 1, activation='sigmoid')(conv9)
  
  model = tf.keras.Model(inputs=inputs, outputs=conv10)
  
  model.compile(optimizer = Adam(lr = 2e-4), loss = 'binary_crossentropy', metrics = [tf.keras.metrics.Accuracy()])
  #model.compile(optimizer=tf.keras.optimizers.Adam(2e-4), loss=combo_loss(alpha=0.2, beta=0.4), metrics=[dice_accuracy])
  #model.compile(optimizer=RMSprop(lr=0.00001), loss=combo_loss, metrics=[dice_accuracy])
  
  if (pretrained_weights):
      model.load_weights(pretrained_weights)
  
  return model
  </code></pre>
  <p>and this is how I save the model:</p>
  <pre><code>model_checkpoint = tf.keras.callbacks.ModelCheckpoint('unet_ThighOuterSurfaceval.hdf5',monitor='val_loss', verbose=1, save_best_only=True)
  model_checkpoint2 = tf.keras.callbacks.ModelCheckpoint('unet_ThighOuterSurface.hdf5', monitor='loss', verbose=1, save_best_only=True)
  
  model = unet_no_dropout()
  history = model.fit(genaug, validation_data=genval, validation_steps=len(genval), steps_per_epoch=len(genaug), epochs=80, callbacks=[model_checkpoint, model_checkpoint2])
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-08-08T02:47:13.287Z
Comments.Text: I met the similar problem and I think the reason is the number of validation data is too small. The small number of data result in contingency——That means the data chosen as the validation set happens to perform well.
Comments.UserDisplayName: jiashu lou
Tags: <tensorflow><keras><image-segmentation><u-net>
Title: Validation accuracy higher than training accurarcy
UserDisplayName: Lis Louise
ThreadText: # Validation accuracy higher than training accurarcy
  
  (Asked by: username_0 on 2020-12-27)
  
  I implemented the unet in TensorFlow for the segmentation of MRI images of the thigh. I noticed I always get a higher validation accuracy by a small gap, independently of the initial split. One example:
  
  [![enter image description here](https://i.sstatic.net/dskC9.png)](https://i.sstatic.net/dskC9.png)
  
  So I researched when this could be possible:
  
  1. When we have an "easy" validation set. I trained it for different initial splitting, all of them showed a higher validation accuracy.
  2. Regularization and augmentation may reduce the training accuracy. I removed the augmentation and dropout regularization and still observed the same gap, the only difference was that it took more epochs to reach convergence.
  3. The last thing I found was that in Keras the training accuracy and loss are averaged over each iteration of the corresponding epoch, while the validation accuracy and loss is calculated from the model at the end of the epoch, which might make the the training loss higher and accuracy lower.
  
  So I thought that if I train and validate on the same set, then I should get the same curve but shifted by one epoch. So I trained only on 2 batches and validated on the same 2 batches (without dropout or augmentation). I still think there is something else happening because they don't look quite the same and at least at the end when the weights are not changing anymore, the training and validation accuracy should be the same (but still the validation accuracy is higher by a small gap). This is the plot:
  
  [![enter image description here](https://i.sstatic.net/9sJlK.png)](https://i.sstatic.net/9sJlK.png)
  [![enter image description here](https://i.sstatic.net/xBUt9.png)](https://i.sstatic.net/xBUt9.png)
  
  Is there anything else that can be increasing the loss values, this is the model I am using:
  
  ```
  def unet_no_dropout(pretrained_weights=None, input_size=(512, 512, 1)):
  inputs = tf.keras.layers.Input(input_size)
  conv1 = tf.keras.layers.Conv2D(64, 3, activation='relu', padding='same', kernel_initializer='he_normal')(inputs)
  conv1 = tf.keras.layers.Conv2D(64, 3, activation='relu', padding='same', kernel_initializer='he_normal')(conv1)
  pool1 = tf.keras.layers.MaxPooling2D(pool_size=(2, 2))(conv1)
  conv2 = tf.keras.layers.Conv2D(128, 3, activation='relu', padding='same', kernel_initializer='he_normal')(pool1)
  conv2 = tf.keras.layers.Conv2D(128, 3, activation='relu', padding='same', kernel_initializer='he_normal')(conv2)
  pool2 = tf.keras.layers.MaxPooling2D(pool_size=(2, 2))(conv2)
  conv3 = tf.keras.layers.Conv2D(256, 3, activation='relu', padding='same', kernel_initializer='he_normal')(pool2)
  conv3 = tf.keras.layers.Conv2D(256, 3, activation='relu', padding='same', kernel_initializer='he_normal')(conv3)
  pool3 = tf.keras.layers.MaxPooling2D(pool_size=(2, 2))(conv3)
  conv4 = tf.keras.layers.Conv2D(512, 3, activation='relu', padding='same', kernel_initializer='he_normal')(pool3)
  conv4 = tf.keras.layers.Conv2D(512, 3, activation='relu', padding='same', kernel_initializer='he_normal')(conv4)
  #drop4 = tf.keras.layers.Dropout(0.5)(conv4)
  pool4 = tf.keras.layers.MaxPooling2D(pool_size=(2, 2))(conv4)
  
  conv5 = tf.keras.layers.Conv2D(1024, 3, activation='relu', padding='same', kernel_initializer='he_normal')(pool4)
  conv5 = tf.keras.layers.Conv2D(1024, 3, activation='relu', padding='same', kernel_initializer='he_normal')(conv5)
  #drop5 = tf.keras.layers.Dropout(0.5)(conv5)
  
  up6 = tf.keras.layers.Conv2D(512, 2, activation='relu', padding='same', kernel_initializer='he_normal')(
      tf.keras.layers.UpSampling2D(size=(2, 2))(conv5))
  merge6 = tf.keras.layers.concatenate([conv4, up6], axis=3)
  #merge6 = tf.keras.layers.concatenate([conv4, up6], axis=3)
  conv6 = tf.keras.layers.Conv2D(512, 3, activation='relu', padding='same', kernel_initializer='he_normal')(merge6)
  conv6 = tf.keras.layers.Conv2D(512, 3, activation='relu', padding='same', kernel_initializer='he_normal')(conv6)
  
  up7 = tf.keras.layers.Conv2D(256, 2, activation='relu', padding='same', kernel_initializer='he_normal')(
      tf.keras.layers.UpSampling2D(size=(2, 2))(conv6))
  merge7 = tf.keras.layers.concatenate([conv3, up7], axis=3)
  conv7 = tf.keras.layers.Conv2D(256, 3, activation='relu', padding='same', kernel_initializer='he_normal')(merge7)
  conv7 = tf.keras.layers.Conv2D(256, 3, activation='relu', padding='same', kernel_initializer='he_normal')(conv7)
  
  up8 = tf.keras.layers.Conv2D(128, 2, activation='relu', padding='same', kernel_initializer='he_normal')(
      tf.keras.layers.UpSampling2D(size=(2, 2))(conv7))
  merge8 = tf.keras.layers.concatenate([conv2, up8], axis=3)
  conv8 = tf.keras.layers.Conv2D(128, 3, activation='relu', padding='same', kernel_initializer='he_normal')(merge8)
  conv8 = tf.keras.layers.Conv2D(128, 3, activation='relu', padding='same', kernel_initializer='he_normal')(conv8)
  
  up9 = tf.keras.layers.Conv2D(64, 2, activation='relu', padding='same', kernel_initializer='he_normal')(
      tf.keras.layers.UpSampling2D(size=(2, 2))(conv8))
  merge9 = tf.keras.layers.concatenate([conv1, up9], axis=3)
  conv9 = tf.keras.layers.Conv2D(64, 3, activation='relu', padding='same', kernel_initializer='he_normal')(merge9)
  conv9 = tf.keras.layers.Conv2D(64, 3, activation='relu', padding='same', kernel_initializer='he_normal')(conv9)
  conv9 = tf.keras.layers.Conv2D(2, 3, activation='relu', padding='same', kernel_initializer='he_normal')(conv9)
  conv10 = tf.keras.layers.Conv2D(1, 1, activation='sigmoid')(conv9)
  
  model = tf.keras.Model(inputs=inputs, outputs=conv10)
  
  model.compile(optimizer = Adam(lr = 2e-4), loss = 'binary_crossentropy', metrics = [tf.keras.metrics.Accuracy()])
  #model.compile(optimizer=tf.keras.optimizers.Adam(2e-4), loss=combo_loss(alpha=0.2, beta=0.4), metrics=[dice_accuracy])
  #model.compile(optimizer=RMSprop(lr=0.00001), loss=combo_loss, metrics=[dice_accuracy])
  
  if (pretrained_weights):
      model.load_weights(pretrained_weights)
  
  return model
  
  ```
  
  and this is how I save the model:
  
  ```
  model_checkpoint = tf.keras.callbacks.ModelCheckpoint('unet_ThighOuterSurfaceval.hdf5',monitor='val_loss', verbose=1, save_best_only=True)
  model_checkpoint2 = tf.keras.callbacks.ModelCheckpoint('unet_ThighOuterSurface.hdf5', monitor='loss', verbose=1, save_best_only=True)
  
  model = unet_no_dropout()
  history = model.fit(genaug, validation_data=genval, validation_steps=len(genval), steps_per_epoch=len(genaug), epochs=80, callbacks=[model_checkpoint, model_checkpoint2])
  
  ```
  
  **Tags:** tensorflow, keras, image-segmentation, u-net
  
  > **Comments:**
  > **username_1**: I met the similar problem and I think the reason is the number of validation data is too small. The small number of data result in contingency——That means the data chosen as the validation set happens to perform well.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2021-09-02. Score: 1 | Accepted answer
  
  I am answering my own question here. The only additional thing that I found was that the average accuracy across a batch of data was slightly higher if the amount of samples in the batch was lower. So the validation set was only 15% of the data, therefore the average accuracy was slightly lower than for 70% of the data. I don't know why the more samples you take the lower the average accuracy, and whether this was a bug in the accuracy calculation or it is the expected behavior. Either way, if you have the same problem, one suggestion is plotting average accuracy vs number of samples and see if this is the reason why you get a lower training accuracy.
  
  > **Comments:**
  > **username_2**: It's probably that smaller sample sizes are more prone to bias. It could just be the train/val split split the validation data into a bias set. Maybe changing the split could give more expected results?  
  
  ---
  
  