Answers.Body: <p>Yes this looks a lot like overfitting. The clue is in the low and slowly decreasing training loss compared to the large increases in validation loss.</p>
  
  <p>One simple fix would be to stop training around epoch 50, taking the best cross validation result to select the most general network at that point. However, anything that works to improve stable generalisation could help here - more training data, more regularisation, simpler model. </p>
  
  <p>The tricky part is finding the best combination for generalisation. Typically if you overuse regularisation to make the NN completely stable, it will lose some accuracy, so you need to run multiple experiments and measure things carefully. As you have a small dataset here and seem capable of running 1000's of epochs, I would suggest k-fold cross validation for improved measurement of cross validation loss.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-07T13:09:32.177Z
Answers.Text: +1 For pointing me to k-fold cross validation which is a new concept for me. Just looked it up and makes intuitive sense. Now I will look for implementing it with Keras. Thanks!
Answers.UserDisplayName: Mike de Klerk
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-10T13:04:04.553Z
Answers.Text: FYI: I implemented the k-fold cross validation. I used a bin size of 5, considered per label. So with 4 labels x 15 examples it resulted in 3 training rounds with each 4 labels x 10 training samples and 4 labels x 5 validation samples. After averaging the training weights and predicting all 60 samples it had 100 accuracy.
Answers.UserDisplayName: Mike de Klerk
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-07T12:10:28.600Z
Answers.LastActivityDate: 2020-03-07T12:10:28.600Z
Answers.UserDisplayName: Neil Slater
Body: <p>I am experimenting with a ConvNet to categorize images taken with a depth camera. So far I have 4 sets of 15 images each. So 4 labels. The original images are 680x880 16-bit grayscale. They are scaled down before feeding it to the ImageDataGenerator to 68x88 RGB (each color channel with equal value). I am using the ImageDataGenerator (IDG) to create more variance on the sets. (The IDG does not seem to be able to handle 16-bit grayscale images, nor 8-bit images well, so hence I converted them to RGB).</p>
  
  <p>I estimate the images to be low on features, compared to regular RGB images, because it represents depth. To get a feel for the images, here are a few down scaled examples:</p>
  
  <p><a href="https://i.sstatic.net/WWmMf.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/WWmMf.jpg" alt="enter image description here"></a>
  <a href="https://i.sstatic.net/en2oi.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/en2oi.jpg" alt="enter image description here"></a>
  <a href="https://i.sstatic.net/Z08jh.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Z08jh.jpg" alt="enter image description here"></a></p>
  
  <p>I let it train 4.096 epochs, to see how that would go.</p>
  
  <p>This is the result of the model and validation loss.
  <a href="https://i.sstatic.net/11LZi.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/11LZi.png" alt="enter image description here"></a></p>
  
  <p>You can see that in the early epochs the validation (test / orange line) loss dips, and then goes up and starts to show big swings. Is this a sign of overfitting?</p>
  
  <p>Here is a zoomed in image of the early epochs.
  <a href="https://i.sstatic.net/z1t7P.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/z1t7P.png" alt="enter image description here"></a></p>
  
  <p>The model loss (train / blue line) reached relatively low values with an accuracy of 1.000. Training again shows repeatedly the same kind of graphs.
  Here are the last epochs.</p>
  
  <pre><code>Epoch 4087/4096
  7/7 [==============================] - 0s 10ms/step - loss: 0.1137 - accuracy: 0.9286 - val_loss: 216.2349 - val_accuracy: 0.7812
  Epoch 4088/4096
  7/7 [==============================] - 0s 10ms/step - loss: 0.0364 - accuracy: 0.9643 - val_loss: 234.9622 - val_accuracy: 0.7812
  Epoch 4089/4096
  7/7 [==============================] - 0s 10ms/step - loss: 0.0041 - accuracy: 1.0000 - val_loss: 232.9797 - val_accuracy: 0.7812
  Epoch 4090/4096
  7/7 [==============================] - 0s 10ms/step - loss: 0.0091 - accuracy: 1.0000 - val_loss: 238.7082 - val_accuracy: 0.7812
  Epoch 4091/4096
  7/7 [==============================] - 0s 10ms/step - loss: 0.0248 - accuracy: 1.0000 - val_loss: 232.4937 - val_accuracy: 0.7812
  Epoch 4092/4096
  7/7 [==============================] - 0s 10ms/step - loss: 0.0335 - accuracy: 0.9643 - val_loss: 273.6542 - val_accuracy: 0.7812
  Epoch 4093/4096
  7/7 [==============================] - 0s 10ms/step - loss: 0.0196 - accuracy: 1.0000 - val_loss: 258.2848 - val_accuracy: 0.7812
  Epoch 4094/4096
  7/7 [==============================] - 0s 10ms/step - loss: 0.0382 - accuracy: 0.9643 - val_loss: 226.6226 - val_accuracy: 0.7812
  Epoch 4095/4096
  7/7 [==============================] - 0s 10ms/step - loss: 0.0018 - accuracy: 1.0000 - val_loss: 226.2943 - val_accuracy: 0.7812
  Epoch 4096/4096
  7/7 [==============================] - 0s 11ms/step - loss: 0.0201 - accuracy: 1.0000 - val_loss: 207.3653 - val_accuracy: 0.7812
  </code></pre>
  
  <p>Not sure if it is required to know the architecture of the neural network to judge whether this is overfitting on this data set. Anyway, here is the setup.</p>
  
  <pre><code>kernelSize = 3
  kernel = (kernelSize, kernelSize)
  
  model = Sequential()
  model.add(Conv2D(16, kernel_size=kernel, padding='same', input_shape=inputShape, activation='relu'))
  model.add(MaxPooling2D(pool_size=(2, 2)))
  
  model.add(Conv2D(32, kernel_size=kernel, padding='same', activation='relu'))
  model.add(MaxPooling2D(pool_size=(2, 2)))
  
  model.add(Conv2D(64, kernel_size=kernel, padding='same', activation='relu'))
  model.add(MaxPooling2D(pool_size=(2, 2)))
  
  model.add(Flatten())  # this converts our 3D feature maps to 1D feature vectors
  model.add(Dense(32, activation='relu'))
  model.add(Dropout(0.2))
  model.add(Dense(nr_of_classes, activation='softmax'))
  
  sgd = tf.keras.optimizers.SGD(lr=learning_rate, decay=1e-6, momentum=0.4, nesterov=True)
  model.compile(loss='categorical_crossentropy', 
              optimizer=sgd,
              metrics=['accuracy'])
  </code></pre>
  
Tags: <convolutional-neural-networks><tensorflow><keras><overfitting><convolution>
Title: Is it a sign of overfitting when validation_loss dips and then goes up with increasingly bigger swings?
UserDisplayName: Mike de Klerk
ThreadText: # Is it a sign of overfitting when validation_loss dips and then goes up with increasingly bigger swings?
  
  (Asked by: username_0 on 2020-03-07)
  
  I am experimenting with a ConvNet to categorize images taken with a depth camera. So far I have 4 sets of 15 images each. So 4 labels. The original images are 680x880 16-bit grayscale. They are scaled down before feeding it to the ImageDataGenerator to 68x88 RGB (each color channel with equal value). I am using the ImageDataGenerator (IDG) to create more variance on the sets. (The IDG does not seem to be able to handle 16-bit grayscale images, nor 8-bit images well, so hence I converted them to RGB).
  I estimate the images to be low on features, compared to regular RGB images, because it represents depth. To get a feel for the images, here are a few down scaled examples:
  
  
  
  I let it train 4.096 epochs, to see how that would go.
  This is the result of the model and validation loss.
  
  You can see that in the early epochs the validation (test / orange line) loss dips, and then goes up and starts to show big swings. Is this a sign of overfitting?
  Here is a zoomed in image of the early epochs.
  
  The model loss (train / blue line) reached relatively low values with an accuracy of 1.000. Training again shows repeatedly the same kind of graphs.
  Here are the last epochs.
  Epoch 4087/4096
  7/7 [==============================] - 0s 10ms/step - loss: 0.1137 - accuracy: 0.9286 - val_loss: 216.2349 - val_accuracy: 0.7812
  Epoch 4088/4096
  7/7 [==============================] - 0s 10ms/step - loss: 0.0364 - accuracy: 0.9643 - val_loss: 234.9622 - val_accuracy: 0.7812
  Epoch 4089/4096
  7/7 [==============================] - 0s 10ms/step - loss: 0.0041 - accuracy: 1.0000 - val_loss: 232.9797 - val_accuracy: 0.7812
  Epoch 4090/4096
  7/7 [==============================] - 0s 10ms/step - loss: 0.0091 - accuracy: 1.0000 - val_loss: 238.7082 - val_accuracy: 0.7812
  Epoch 4091/4096
  7/7 [==============================] - 0s 10ms/step - loss: 0.0248 - accuracy: 1.0000 - val_loss: 232.4937 - val_accuracy: 0.7812
  Epoch 4092/4096
  7/7 [==============================] - 0s 10ms/step - loss: 0.0335 - accuracy: 0.9643 - val_loss: 273.6542 - val_accuracy: 0.7812
  Epoch 4093/4096
  7/7 [==============================] - 0s 10ms/step - loss: 0.0196 - accuracy: 1.0000 - val_loss: 258.2848 - val_accuracy: 0.7812
  Epoch 4094/4096
  7/7 [==============================] - 0s 10ms/step - loss: 0.0382 - accuracy: 0.9643 - val_loss: 226.6226 - val_accuracy: 0.7812
  Epoch 4095/4096
  7/7 [==============================] - 0s 10ms/step - loss: 0.0018 - accuracy: 1.0000 - val_loss: 226.2943 - val_accuracy: 0.7812
  Epoch 4096/4096
  7/7 [==============================] - 0s 11ms/step - loss: 0.0201 - accuracy: 1.0000 - val_loss: 207.3653 - val_accuracy: 0.7812
  
  Not sure if it is required to know the architecture of the neural network to judge whether this is overfitting on this data set. Anyway, here is the setup.
  kernelSize = 3
  kernel = (kernelSize, kernelSize)
  
  model = Sequential()
  model.add(Conv2D(16, kernel_size=kernel, padding='same', input_shape=inputShape, activation='relu'))
  model.add(MaxPooling2D(pool_size=(2, 2)))
  
  model.add(Conv2D(32, kernel_size=kernel, padding='same', activation='relu'))
  model.add(MaxPooling2D(pool_size=(2, 2)))
  
  model.add(Conv2D(64, kernel_size=kernel, padding='same', activation='relu'))
  model.add(MaxPooling2D(pool_size=(2, 2)))
  
  model.add(Flatten())  # this converts our 3D feature maps to 1D feature vectors
  model.add(Dense(32, activation='relu'))
  model.add(Dropout(0.2))
  model.add(Dense(nr_of_classes, activation='softmax'))
  
  sgd = tf.keras.optimizers.SGD(lr=learning_rate, decay=1e-6, momentum=0.4, nesterov=True)
  model.compile(loss='categorical_crossentropy', 
              optimizer=sgd,
              metrics=['accuracy'])
  
  **Tags:** convolutional-neural-networks, tensorflow, keras, overfitting, convolution
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-03-07. Score: 2 | Accepted answer
  
  Yes this looks a lot like overfitting. The clue is in the low and slowly decreasing training loss compared to the large increases in validation loss.
  
  One simple fix would be to stop training around epoch 50, taking the best cross validation result to select the most general network at that point. However, anything that works to improve stable generalisation could help here - more training data, more regularisation, simpler model.
  
  The tricky part is finding the best combination for generalisation. Typically if you overuse regularisation to make the NN completely stable, it will lose some accuracy, so you need to run multiple experiments and measure things carefully. As you have a small dataset here and seem capable of running 1000's of epochs, I would suggest k-fold cross validation for improved measurement of cross validation loss.
  
  > **Comments:**
  > **username_0**: +1 For pointing me to k-fold cross validation which is a new concept for me. Just looked it up and makes intuitive sense. Now I will look for implementing it with Keras. Thanks!  
  > **username_0**: FYI: I implemented the k-fold cross validation. I used a bin size of 5, considered per label. So with 4 labels x 15 examples it resulted in 3 training rounds with each 4 labels x 10 training samples and 4 labels x 5 validation samples. After averaging the training weights and predicting all 60 samples it had 100 accuracy.  
  
  ---
  
  