Answers.Body: <p>I do not understand why you say that your model is overfitting. An overfit occurs when the validation loss start increasing after diminishing. Here it seems that your model has reaches its potential and cannot improve anymore. What I would recommend here is to make your model bigger: add filters, increase the depth. Also consider trying transfer learning; it is a common base to all tasks.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-18T23:30:03.710Z
Answers.Text: Ok I will try working with a bigger model. Thanks for your inputs. I also updated my recent graph in the question above considering the inputs given by @spb and got these results. Do you think its overfitting or the model is basically reaching its potential?
Answers.UserDisplayName: Yishu Malhotra
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-19T13:24:02.090Z
Answers.Text: Still, I don't see the shadow of an overfitting. However, I checked the dataset that you use, and it says to contain 4795 images. I think that training a model from scratch on a small dataset like this will not give you incredible results. Transfer learning can help a lot with that, combined with data augmentation.
Answers.UserDisplayName: bguetarni
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-19T14:36:35.017Z
Answers.Text: Any recommendations which weights I shall use for transfer learning?
Answers.UserDisplayName: Yishu Malhotra
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-19T14:46:21.397Z
Answers.Text: The U-Net architecture is basically an encoder and a decoder with skip connections between them. you can use any pre-trained deep ConvNet for the encoder part and then build your decoder accordingly with transposed convolutions; and don't forget to add skip connections - very important.
  [Here](https://www.tensorflow.org/api_docs/python/tf/keras/applications) you have the pre-trained models available in tensorflow that you can use for the encoder.
Answers.UserDisplayName: bguetarni
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-18T13:00:09.407Z
Answers.LastActivityDate: 2021-05-18T13:00:09.407Z
Answers.UserDisplayName: bguetarni
Answers.Body: <p>Data augmentations is usually done on the fly during training, meaning before each you apply the random augmentation for the entire dataset, because of the randomness there will be different transformation of the same image in each epoch.</p>
  <p>Shuffle the dataset before batching in each epoch, so that each epoch will not have minibatch of same images, which will reduce overfitting. Learning rate usually 1e-4 works fine for me.</p>
  <p>Your UNet is not wide enough, why are you using only 16 filters in first conv block, original UNet paper had 64 filters in first conv block. Also you have only one convolution block in each layer, why? original unet has 2 conv blocks in each layer. I suggest you to try with unet given in here
  <a href="https://github.com/zhixuhao/unet/blob/master/model.py" rel="nofollow noreferrer">https://github.com/zhixuhao/unet/blob/master/model.py</a></p>
  <p>Dice loss is usually prefered for segmentation, check code here</p>
  <pre><code>from keras import backend as K
  
  def dice_score(y_true, y_pred, smooth=1e-7):
  
      intersection = K.sum(K.abs(y_true * y_pred), axis=-1)
      return (2. * intersection + smooth) / (K.sum(K.square(y_true),-1) + K.sum(K.square(y_pred),-1) + smooth)
  
  def dice_loss(y_true, y_pred):
      return 1-dice_score(y_true, y_pred)
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-15T12:42:28.907Z
Answers.Text: Thanks for your in detail answer. Just to clear some things, so random augmentations should be applied before each epoch on the entire dataset, right? 
  Also I updated the code in the post above, I am using two convolution layers, conv2d_block is actually a function i made for each layer, i have put that code above as well for your reference.
Answers.UserDisplayName: Yishu Malhotra
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-15T12:47:39.093Z
Answers.Text: Yes, apply random augmentations before each epoch on the entire dataset.
Answers.UserDisplayName: spb
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-15T12:48:57.720Z
Answers.Text: What about number of filters in the first layer, did you try changing to 64? Also try removing dropouts in contracting path, it worked well for me.
Answers.UserDisplayName: spb
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-15T12:53:11.017Z
Answers.Text: Yes, I am not using dropouts they are commented, so you used dropouts in the expanding path?
  I am going to try changing filters to 64 now and get back to you.
Answers.UserDisplayName: Yishu Malhotra
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-15T12:56:33.797Z
Answers.Text: Yes, dropout(0.5) in expanding path reduced overfitting
Answers.UserDisplayName: spb
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-14T13:09:37.400Z
Answers.LastActivityDate: 2021-05-14T13:09:37.400Z
Answers.UserDisplayName: spb
Body: <p>I am working on a project where I am trying to detect and localize forgeries in images. I am using the CASIA v2 dataset and using Unet model for the task. I have the binary masks of all the images in the CASIA v2 dataset. The metric I am using for the model are F1 score.</p>
  <p>The issue with the model is that it is highly overfitting, the validation loss plateaus up.<a href="https://i.sstatic.net/43mV9.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/43mV9.png" alt="Validation loss and validation F1 score plateau, model overfitting" /></a></p>
  <p>Batch size is 128 and Learning rate is 0.000001. Image size is 128 x 128.</p>
  <p>Updated graph for batch size 16 with the changes mentioned by @spb is as follows:
  <a href="https://i.sstatic.net/WXQhd.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/WXQhd.png" alt="enter image description here" /></a></p>
  <p>I have also tried using Learning rate scheduler to decrease the learning rate(starting with high learning rate) on plateaus but that didn't help much.</p>
  <p>I am also using the package Albumentations for data augmentation of both the images and its masks. I load the images and the masks and then apply the augmentations and save the augmented images and masks in a separate arrays and finally extend the original images and masks with the augmented images and masks. So technically I have original plus the augmented images and masks that I use for training the model. The augmentations I am using are:</p>
  <pre><code>Augment = A.Compose([
  A.VerticalFlip(p=0.5),
  A.RandomRotate90(p=0.5),
  A.HorizontalFlip(p = 0.5)
  ])
  </code></pre>
  <p>I have split the dataset into 70% Training, 20% Validation and 10% for testing.
  Here is a snippet of my model. <strong>Updated Code below</strong></p>
  <pre><code>def conv2d_block(input_tensor, n_filters, kernel_size = 3, batchnorm = True):
  &quot;&quot;&quot;Function to add 2 convolutional layers with the parameters passed to it&quot;&quot;&quot;
  # first layer
  x = Conv2D(filters = n_filters, kernel_size = (kernel_size, kernel_size),\
            kernel_initializer = 'he_normal', padding = 'same')(input_tensor)
  if batchnorm:
      x = BatchNormalization()(x)
  x = Activation('relu')(x)
  
  # second layer
  x = Conv2D(filters = n_filters, kernel_size = (kernel_size, kernel_size),\
            kernel_initializer = 'he_normal', padding = 'same')(input_tensor)
  if batchnorm:
      x = BatchNormalization()(x)
  x = Activation('relu')(x)
  
  return x
  
  def get_unet(input_img, n_filters = 16, dropout = 0.1, batchnorm = True):
  &quot;&quot;&quot;Function to define the UNET Model&quot;&quot;&quot;
  # Contracting Path
  c1 = conv2d_block(input_img, n_filters * 1, kernel_size = 3, batchnorm = batchnorm)
  p1 = MaxPooling2D((2, 2))(c1)
  #p1 = Dropout(dropout)(p1)
  
  c2 = conv2d_block(p1, n_filters * 2, kernel_size = 3, batchnorm = batchnorm)
  p2 = MaxPooling2D((2, 2))(c2)
  #p2 = Dropout(dropout)(p2)
  
  c3 = conv2d_block(p2, n_filters * 4, kernel_size = 3, batchnorm = batchnorm)
  p3 = MaxPooling2D((2, 2))(c3)
  #p3 = Dropout(dropout)(p3)
  
  c4 = conv2d_block(p3, n_filters * 8, kernel_size = 3, batchnorm = batchnorm)
  p4 = MaxPooling2D((2, 2))(c4)
  #p4 = Dropout(dropout)(p4)
  
  c5 = conv2d_block(p4, n_filters * 16, kernel_size = 3, batchnorm = batchnorm)
  p5 = MaxPooling2D((2, 2))(c5)
  #p5 = Dropout(dropout)(p5)
  
  c6 = conv2d_block(p5, n_filters = n_filters * 32, kernel_size = 3, batchnorm = batchnorm)
  
  # Expansive Path
  u7 = Conv2DTranspose(n_filters * 16, (3, 3), strides = (2, 2), padding = 'same')(c6)
  u7 = concatenate([u7, c5])
  u7 = Dropout(dropout)(u7)
  c7 = conv2d_block(u7, n_filters * 16, kernel_size = 3, batchnorm = batchnorm)
  
  u8 = Conv2DTranspose(n_filters * 8, (3, 3), strides = (2, 2), padding = 'same')(c7)
  u8 = concatenate([u8, c4])
  u8 = Dropout(dropout)(u8)
  c8 = conv2d_block(u8, n_filters * 8, kernel_size = 3, batchnorm = batchnorm)
  
  u9 = Conv2DTranspose(n_filters * 4, (3, 3), strides = (2, 2), padding = 'same')(c8)
  u9 = concatenate([u9, c3])
  u9 = Dropout(dropout)(u9)
  c9 = conv2d_block(u9, n_filters * 4, kernel_size = 3, batchnorm = batchnorm)
  
  u10 = Conv2DTranspose(n_filters * 2, (3, 3), strides = (2, 2), padding = 'same')(c9)
  u10 = concatenate([u10, c2])
  u10 = Dropout(dropout)(u10)
  c10 = conv2d_block(u10, n_filters * 2, kernel_size = 3, batchnorm = batchnorm)
  
  u11 = Conv2DTranspose(n_filters * 1, (3, 3), strides = (2, 2), padding = 'same')(c10)
  u11 = concatenate([u11, c1])
  u11 = Dropout(dropout)(u11)
  c11 = conv2d_block(u11, n_filters * 1, kernel_size = 3, batchnorm = batchnorm)
  
  outputs = Conv2D(1, (1, 1), activation='sigmoid')(c11)
  model = Model(inputs=[input_img], outputs=[outputs])
  return model
  </code></pre>
  <p>Currently I am not using the dropout as it leads to higher validation loss plateaus in my case.</p>
  <p>The F1 score and  F1 loss I am calculating are as follows</p>
  <pre><code>def f1(y_true, y_pred):
  
  y_pred = K.round(y_pred)
  tp = K.sum(K.cast(y_true*y_pred, 'float'), axis=0)
  tn = K.sum(K.cast((1-y_true)*(1-y_pred), 'float'), axis=0)
  fp = K.sum(K.cast((1-y_true)*y_pred, 'float'), axis=0)
  fn = K.sum(K.cast(y_true*(1-y_pred), 'float'), axis=0)
  
  p = tp / (tp + fp + K.epsilon())
  r = tp / (tp + fn + K.epsilon())
  
  f1 = 2*p*r / (p+r+K.epsilon())
  f1 = tf.where(tf.is_nan(f1), tf.zeros_like(f1), f1)
  return K.mean(f1)
  
  def f1_loss(y_true, y_pred):
  
  tp = K.sum(K.cast(y_true*y_pred, 'float'), axis=0)
  tn = K.sum(K.cast((1-y_true)*(1-y_pred), 'float'), axis=0)
  fp = K.sum(K.cast((1-y_true)*y_pred, 'float'), axis=0)
  fn = K.sum(K.cast(y_true*(1-y_pred), 'float'), axis=0)
  
  p = tp / (tp + fp + K.epsilon())
  r = tp / (tp + fn + K.epsilon())
  
  f1 = 2*p*r / (p+r+K.epsilon())
  f1 = tf.where(tf.is_nan(f1), tf.zeros_like(f1), f1)
  return 1 - K.mean(f1)
  </code></pre>
  <p>I have also tried using other losses like <strong>focal_tversky</strong> but have a similar result.</p>
  <p>What can be the issue  and how can I solve it?</p>
  <p>Is it</p>
  <ol>
  <li>Issue with my data like presence of outliers</li>
  <li>Model related issue</li>
  <li>Batch size and Learning rate related issue</li>
  <li>Or anything else?</li>
  </ol>
  <p>Please your help in this regard is really appreciated as I really need to solve it soon.</p>
  
Tags: <deep-learning><computer-vision><tensorflow><image-segmentation><u-net>
Title: Unet Overfitting for binary segmentation of fake images
UserDisplayName: Yishu Malhotra
ThreadText: # Unet Overfitting for binary segmentation of fake images
  
  (Asked by: username_0 on 2021-05-13)
  
  I am working on a project where I am trying to detect and localize forgeries in images. I am using the CASIA v2 dataset and using Unet model for the task. I have the binary masks of all the images in the CASIA v2 dataset. The metric I am using for the model are F1 score.
  
  The issue with the model is that it is highly overfitting, the validation loss plateaus up.[![Validation loss and validation F1 score plateau, model overfitting](https://i.sstatic.net/43mV9.png)](https://i.sstatic.net/43mV9.png)
  
  Batch size is 128 and Learning rate is 0.000001. Image size is 128 x 128.
  
  Updated graph for batch size 16 with the changes mentioned by @username_2 is as follows:
  [![enter image description here](https://i.sstatic.net/WXQhd.png)](https://i.sstatic.net/WXQhd.png)
  
  I have also tried using Learning rate scheduler to decrease the learning rate(starting with high learning rate) on plateaus but that didn't help much.
  
  I am also using the package Albumentations for data augmentation of both the images and its masks. I load the images and the masks and then apply the augmentations and save the augmented images and masks in a separate arrays and finally extend the original images and masks with the augmented images and masks. So technically I have original plus the augmented images and masks that I use for training the model. The augmentations I am using are:
  
  ```
  Augment = A.Compose([
  A.VerticalFlip(p=0.5),
  A.RandomRotate90(p=0.5),
  A.HorizontalFlip(p = 0.5)
  ])
  
  ```
  
  I have split the dataset into 70% Training, 20% Validation and 10% for testing.
  Here is a snippet of my model. **Updated Code below**
  
  ```
  def conv2d_block(input_tensor, n_filters, kernel_size = 3, batchnorm = True):
  """Function to add 2 convolutional layers with the parameters passed to it"""
  # first layer
  x = Conv2D(filters = n_filters, kernel_size = (kernel_size, kernel_size),\
            kernel_initializer = 'he_normal', padding = 'same')(input_tensor)
  if batchnorm:
      x = BatchNormalization()(x)
  x = Activation('relu')(x)
  
  # second layer
  x = Conv2D(filters = n_filters, kernel_size = (kernel_size, kernel_size),\
            kernel_initializer = 'he_normal', padding = 'same')(input_tensor)
  if batchnorm:
      x = BatchNormalization()(x)
  x = Activation('relu')(x)
  
  return x
  
  def get_unet(input_img, n_filters = 16, dropout = 0.1, batchnorm = True):
  """Function to define the UNET Model"""
  # Contracting Path
  c1 = conv2d_block(input_img, n_filters * 1, kernel_size = 3, batchnorm = batchnorm)
  p1 = MaxPooling2D((2, 2))(c1)
  #p1 = Dropout(dropout)(p1)
  
  c2 = conv2d_block(p1, n_filters * 2, kernel_size = 3, batchnorm = batchnorm)
  p2 = MaxPooling2D((2, 2))(c2)
  #p2 = Dropout(dropout)(p2)
  
  c3 = conv2d_block(p2, n_filters * 4, kernel_size = 3, batchnorm = batchnorm)
  p3 = MaxPooling2D((2, 2))(c3)
  #p3 = Dropout(dropout)(p3)
  
  c4 = conv2d_block(p3, n_filters * 8, kernel_size = 3, batchnorm = batchnorm)
  p4 = MaxPooling2D((2, 2))(c4)
  #p4 = Dropout(dropout)(p4)
  
  c5 = conv2d_block(p4, n_filters * 16, kernel_size = 3, batchnorm = batchnorm)
  p5 = MaxPooling2D((2, 2))(c5)
  #p5 = Dropout(dropout)(p5)
  
  c6 = conv2d_block(p5, n_filters = n_filters * 32, kernel_size = 3, batchnorm = batchnorm)
  
  # Expansive Path
  u7 = Conv2DTranspose(n_filters * 16, (3, 3), strides = (2, 2), padding = 'same')(c6)
  u7 = concatenate([u7, c5])
  u7 = Dropout(dropout)(u7)
  c7 = conv2d_block(u7, n_filters * 16, kernel_size = 3, batchnorm = batchnorm)
  
  u8 = Conv2DTranspose(n_filters * 8, (3, 3), strides = (2, 2), padding = 'same')(c7)
  u8 = concatenate([u8, c4])
  u8 = Dropout(dropout)(u8)
  c8 = conv2d_block(u8, n_filters * 8, kernel_size = 3, batchnorm = batchnorm)
  
  u9 = Conv2DTranspose(n_filters * 4, (3, 3), strides = (2, 2), padding = 'same')(c8)
  u9 = concatenate([u9, c3])
  u9 = Dropout(dropout)(u9)
  c9 = conv2d_block(u9, n_filters * 4, kernel_size = 3, batchnorm = batchnorm)
  
  u10 = Conv2DTranspose(n_filters * 2, (3, 3), strides = (2, 2), padding = 'same')(c9)
  u10 = concatenate([u10, c2])
  u10 = Dropout(dropout)(u10)
  c10 = conv2d_block(u10, n_filters * 2, kernel_size = 3, batchnorm = batchnorm)
  
  u11 = Conv2DTranspose(n_filters * 1, (3, 3), strides = (2, 2), padding = 'same')(c10)
  u11 = concatenate([u11, c1])
  u11 = Dropout(dropout)(u11)
  c11 = conv2d_block(u11, n_filters * 1, kernel_size = 3, batchnorm = batchnorm)
  
  outputs = Conv2D(1, (1, 1), activation='sigmoid')(c11)
  model = Model(inputs=[input_img], outputs=[outputs])
  return model
  
  ```
  
  Currently I am not using the dropout as it leads to higher validation loss plateaus in my case.
  
  The F1 score and F1 loss I am calculating are as follows
  
  ```
  def f1(y_true, y_pred):
  
  y_pred = K.round(y_pred)
  tp = K.sum(K.cast(y_true*y_pred, 'float'), axis=0)
  tn = K.sum(K.cast((1-y_true)*(1-y_pred), 'float'), axis=0)
  fp = K.sum(K.cast((1-y_true)*y_pred, 'float'), axis=0)
  fn = K.sum(K.cast(y_true*(1-y_pred), 'float'), axis=0)
  
  p = tp / (tp + fp + K.epsilon())
  r = tp / (tp + fn + K.epsilon())
  
  f1 = 2*p*r / (p+r+K.epsilon())
  f1 = tf.where(tf.is_nan(f1), tf.zeros_like(f1), f1)
  return K.mean(f1)
  
  def f1_loss(y_true, y_pred):
  
  tp = K.sum(K.cast(y_true*y_pred, 'float'), axis=0)
  tn = K.sum(K.cast((1-y_true)*(1-y_pred), 'float'), axis=0)
  fp = K.sum(K.cast((1-y_true)*y_pred, 'float'), axis=0)
  fn = K.sum(K.cast(y_true*(1-y_pred), 'float'), axis=0)
  
  p = tp / (tp + fp + K.epsilon())
  r = tp / (tp + fn + K.epsilon())
  
  f1 = 2*p*r / (p+r+K.epsilon())
  f1 = tf.where(tf.is_nan(f1), tf.zeros_like(f1), f1)
  return 1 - K.mean(f1)
  
  ```
  
  I have also tried using other losses like **focal\_tversky** but have a similar result.
  
  What can be the issue and how can I solve it?
  
  Is it
  
  1. Issue with my data like presence of outliers
  2. Model related issue
  3. Batch size and Learning rate related issue
  4. Or anything else?
  
  Please your help in this regard is really appreciated as I really need to solve it soon.
  
  **Tags:** deep-learning, computer-vision, tensorflow, image-segmentation, u-net
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2021-05-18. Score: 1
  
  I do not understand why you say that your model is overfitting. An overfit occurs when the validation loss start increasing after diminishing. Here it seems that your model has reaches its potential and cannot improve anymore. What I would recommend here is to make your model bigger: add filters, increase the depth. Also consider trying transfer learning; it is a common base to all tasks.
  
  > **Comments:**
  > **username_0**: Ok I will try working with a bigger model. Thanks for your inputs. I also updated my recent graph in the question above considering the inputs given by @username_2 and got these results. Do you think its overfitting or the model is basically reaching its potential?  
  > **username_1**: Still, I don't see the shadow of an overfitting. However, I checked the dataset that you use, and it says to contain 4795 images. I think that training a model from scratch on a small dataset like this will not give you incredible results. Transfer learning can help a lot with that, combined with data augmentation.  
  > **username_0**: Any recommendations which weights I shall use for transfer learning?  
  > **username_1**: The U-Net architecture is basically an encoder and a decoder with skip connections between them. you can use any pre-trained deep ConvNet for the encoder part and then build your decoder accordingly with transposed convolutions; and don't forget to add skip connections - very important. [Here](https://www.tensorflow.org/api_docs/python/tf/keras/applications) you have the pre-trained models available in tensorflow that you can use for the encoder.  
  
  ---
  
  ### Answer by username_2 on 2021-05-14. Score: 1
  
  Data augmentations is usually done on the fly during training, meaning before each you apply the random augmentation for the entire dataset, because of the randomness there will be different transformation of the same image in each epoch.
  
  Shuffle the dataset before batching in each epoch, so that each epoch will not have minibatch of same images, which will reduce overfitting. Learning rate usually 1e-4 works fine for me.
  
  Your UNet is not wide enough, why are you using only 16 filters in first conv block, original UNet paper had 64 filters in first conv block. Also you have only one convolution block in each layer, why? original unet has 2 conv blocks in each layer. I suggest you to try with unet given in here
  <https://github.com/zhixuhao/unet/blob/master/model.py>
  
  Dice loss is usually prefered for segmentation, check code here
  
  ```
  from keras import backend as K
  
  def dice_score(y_true, y_pred, smooth=1e-7):
  
      intersection = K.sum(K.abs(y_true * y_pred), axis=-1)
      return (2. * intersection + smooth) / (K.sum(K.square(y_true),-1) + K.sum(K.square(y_pred),-1) + smooth)
  
  def dice_loss(y_true, y_pred):
      return 1-dice_score(y_true, y_pred)
  
  ```
  
  > **Comments:**
  > **username_0**: Thanks for your in detail answer. Just to clear some things, so random augmentations should be applied before each epoch on the entire dataset, right?  Also I updated the code in the post above, I am using two convolution layers, conv2d_block is actually a function i made for each layer, i have put that code above as well for your reference.  
  > **username_2**: Yes, apply random augmentations before each epoch on the entire dataset.  
  > **username_2**: What about number of filters in the first layer, did you try changing to 64? Also try removing dropouts in contracting path, it worked well for me.  
  > **username_0**: Yes, I am not using dropouts they are commented, so you used dropouts in the expanding path? I am going to try changing filters to 64 now and get back to you.  
  > **username_2**: Yes, dropout(0.5) in expanding path reduced overfitting  
  
  ---
  
  