title: Deep learning CNN model not learning
body: <p>I am working on a image classification project and my model doesn't seem to train properly.</p>
  <p>My dataset is made of 4000 images each with a shape of (120,120,3).
  Test set represents 20% of the total dataset.
  All images have been correctly labeled.
  The images are normalized and one-hot encoded. For now I use only two targets, but I will add one more one I start getting decent results.
  I use a batch size of 16
  I want to use a CNN model.</p>
  <p>My current model :</p>
  <pre><code>model = keras.models.Sequential()
  model.add(Conv2D(filters=16, kernel_size=(6,6), input_shape=(IMG_SIZE,IMG_SIZE,3), activation='relu',))
  model.add(MaxPool2D(pool_size=(2,2)))
  model.add(Dropout(0.2))
  
  model.add(Conv2D(filters=32, kernel_size=(5,5), activation='relu',))
  model.add(MaxPool2D(pool_size=(2,2)))
  model.add(Dropout(0.2))
  model.add(Conv2D(filters=64, kernel_size=(4,4), activation='relu',))
  model.add(MaxPool2D(pool_size=(2,2)))
  model.add(Dropout(0.2))
  model.add(Conv2D(filters=128, kernel_size=(3,3), activation='relu',))
  model.add(MaxPool2D(pool_size=(2,2)))
  model.add(Dropout(0.2))
  model.add(Conv2D(filters=256, kernel_size=(2,2), activation='relu',))
  model.add(MaxPool2D(pool_size=(2,2)))
  model.add(Dropout(0.2))
  
  
  model.add(Flatten())
  
  
  model.add(Dense(64, activation='relu'))
  model.add(Dropout(0.5))
  model.add(Dense(64, activation='relu'))
  model.add(Dropout(0.5))
  model.add(Dense(2, activation='softmax'))
  
  model.compile(optimizer='nadam',
               loss='categorical_crossentropy',
              metrics=['accuracy'])
  
  model.summary()
  </code></pre>
  <p>model summary gives :
  Total params: 273,330
  Trainable params: 273,330
  Non-trainable params: 0</p>
  <pre><code>from tensorflow.keras.callbacks import EarlyStopping
  early_stop = EarlyStopping(monitor='val_loss',patience=10)
  
  
  
  history = model.fit(x_train_sample, y_train_sample,
                     batch_size = BATCH_SIZE,
                     epochs = EPOCHS,
                     verbose = 1,
                     validation_data = (x_test, y_test)
                     ,callbacks=[early_stop,PlotLossesKeras()])
  
  </code></pre>
  <p>When I run my model for 30 epochs, earlystopping triggers.</p>
  <pre><code>Epoch 1/30
  43/43 [==============================] - 9s 205ms/step - loss: 0.1109 - accuracy: 0.9531 - val_loss: 0.5259 - val_accuracy: 0.8397
  Epoch 2/30
  43/43 [==============================] - 10s 231ms/step - loss: 0.0812 - accuracy: 0.9692 - val_loss: 0.5793 - val_accuracy: 0.8355
  Epoch 3/30
  43/43 [==============================] - 9s 219ms/step - loss: 0.1000 - accuracy: 0.9721 - val_loss: 0.5367 - val_accuracy: 0.8547
  Epoch 4/30
  43/43 [==============================] - 9s 209ms/step - loss: 0.0694 - accuracy: 0.9707 - val_loss: 0.6101 - val_accuracy: 0.8269
  Epoch 5/30
  43/43 [==============================] - 9s 203ms/step - loss: 0.0891 - accuracy: 0.9633 - val_loss: 0.6116 - val_accuracy: 0.8419
  Epoch 6/30
  43/43 [==============================] - 9s 210ms/step - loss: 0.0567 - accuracy: 0.9765 - val_loss: 0.4833 - val_accuracy: 0.8419
  Epoch 7/30
  43/43 [==============================] - 9s 218ms/step - loss: 0.0312 - accuracy: 0.9897 - val_loss: 1.4513 - val_accuracy: 0.8034
  Epoch 8/30
  43/43 [==============================] - 9s 213ms/step - loss: 0.0820 - accuracy: 0.9707 - val_loss: 0.5821 - val_accuracy: 0.8248
  Epoch 9/30
  43/43 [==============================] - 9s 222ms/step - loss: 0.0513 - accuracy: 0.9897 - val_loss: 0.8516 - val_accuracy: 0.8462
  Epoch 10/30
  43/43 [==============================] - 11s 246ms/step - loss: 0.0442 - accuracy: 0.9853 - val_loss: 0.7927 - val_accuracy: 0.8397
  Epoch 11/30
  43/43 [==============================] - 10s 222ms/step - loss: 0.0356 - accuracy: 0.9897 - val_loss: 0.7730 - val_accuracy: 0.8141
  Epoch 12/30
  43/43 [==============================] - 10s 232ms/step - loss: 0.0309 - accuracy: 0.9824 - val_loss: 0.9528 - val_accuracy: 0.8226
  Epoch 13/30
  43/43 [==============================] - 9s 220ms/step - loss: 0.0424 - accuracy: 0.9839 - val_loss: 1.2109 - val_accuracy: 0.8013
  Epoch 14/30
  43/43 [==============================] - 10s 228ms/step - loss: 0.0645 - accuracy: 0.9824 - val_loss: 0.5308 - val_accuracy: 0.8547
  Epoch 15/30
  43/43 [==============================] - 11s 259ms/step - loss: 0.0293 - accuracy: 0.9927 - val_loss: 0.9271 - val_accuracy: 0.8333
  Epoch 16/30
  43/43 [==============================] - 9s 217ms/step - loss: 0.0430 - accuracy: 0.9795 - val_loss: 0.6687 - val_accuracy: 0.8483
  
  </code></pre>
  <p>I have tried many different model architectures, changing number of layers, kernel size etc... I can't seem to figure out what is going wrong.</p>