Body: <p>I have CNN model to classify 2 classes. (Yes or No)
  I use categorical_crossentropy loss and softmax activation at the end. 
  For input I use image with all 3 channels, for output I use One hot encoded vector ([0,1] or [1,0])</p>
  
  <p>I have function that guaranty me, that each batch I have same number of one and another class, so the classes are not unevenly represented. </p>
  
  <p>What happened when I train the model is that I am stuck at same loss while trening,...</p>
  
  <p>I assume that model predict always same class and half in batch has loss 0 half of them max, so that bring it to 8 all the time,...</p>
  
  <p>What could went wrong?</p>
  
  <p>The network is something like this : </p>
  
  <pre><code>x = Conv2D(16, (3, 3), padding='same')(input_img)
  x = LeakyReLU(0.1)(x)
  x = Conv2D(32 , (3, 3), padding='same')(x)
  x = LeakyReLU(0.1)(x)
  x = MaxPooling2D((2, 2))(x)
  x = Dropout(0.25)(x)
  x = Conv2D(32 , (3, 3), padding='same')(x)
  x = LeakyReLU(0.1)(x)
  x = Conv2D(48 , (3, 3), padding='same')(x)
  x = LeakyReLU(0.1)(x)
  x = MaxPooling2D((2, 2))(x)
  x = Dropout(0.25)(x)
  
  x = Flatten()(x)
  x = Dense(4096)(x)
  x = LeakyReLU(0.1)(x)
  x = Dropout(0.5)(x)
  x = Dense(2048)(x)
  x = LeakyReLU(0.1)(x)
  x = Dropout(0.5)(x)
  out = Dense(2, activation='softmax', name='table')(x)
  
  model = Model(input_img, out)
  model.compile(optimizer='adam', loss= 'categorical_crossentropy')
  </code></pre>
  
  <p><strong>Training Loss:</strong> </p>
  
  <p><a href="https://i.sstatic.net/xLsTb.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/xLsTb.png" alt="enter image description here"></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-09-29T20:13:53.720Z
Comments.Text: I figure out, that sometimes bigger batch size helps : from 32 to 256,... But I would appreciate some theoretical and practical advice to solve this king of errors...
Comments.UserDisplayName: Marko Zadravec
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-09-30T06:00:07.567Z
Comments.Text: There are many things that you could be doing wrong. 1. Are you sure your initialisation is correct? An unchanging error is sometimes a sign that an entire layer is dead, hence no ability to change the error. 2. Are you sure your backprop is updating everything? 3. Are you sure that you are definitely using a one hot vector as the output 4. If you try training your CNN on something else, is it able to learn? 5. Make sure your cross entropy loss function is correct. I can almost guarantee a behaviour like this is caused by an error in the final 2 or so layers of the network.
Comments.UserDisplayName: Recessive
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-10-01T05:18:24.670Z
Comments.Text: @Recessive : 1. Initialisation is handle by Adam,.. and I use LeakyReLU not to produce 0 output 2. Yes 3. Yes I always use [0,1] or [1,0]. 4. I need to check this, but I think so (Usually if I increase batch size it train better, but I need to increase it to 256 or more,...).  5. I use default Keras loss function implementation. Regarding last layers : They are normal dense layers with LeakyReLU activations. What did you have in mind as a potential  error?
Comments.UserDisplayName: Marko Zadravec
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-10-01T05:21:40.837Z
Comments.Text: I update my question, to give you some ideas how network look like,...
Comments.UserDisplayName: Marko Zadravec
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-10-01T12:23:04.610Z
Comments.Text: Try taking the dropout out. What's the input dimensions of the image?
Comments.UserDisplayName: Recessive
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-10-01T13:16:13.357Z
Comments.Text: Is this the training loss or validation loss ? Did you try to print out the predictions made by your network ?
Comments.UserDisplayName: Joseph Budin
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-10-02T04:38:02.857Z
Comments.Text: Even with dropouts, I get the same result. This is training loss (I update question). The model always predict only one class. This is the reason (I guess for constant loss, because half of the batch has zero loss, half max loss because it always outputs the same class).
Comments.UserDisplayName: Marko Zadravec
Tags: <convolutional-neural-networks><objective-functions><categorical-data>
Title: CNN clasification model loss stuck at same value
UserDisplayName: Marko Zadravec
ThreadText: # CNN clasification model loss stuck at same value
  
  (Asked by: username_0 on 2019-09-26)
  
  I have CNN model to classify 2 classes. (Yes or No)
  I use categorical_crossentropy loss and softmax activation at the end. 
  For input I use image with all 3 channels, for output I use One hot encoded vector ([0,1] or [1,0])
  I have function that guaranty me, that each batch I have same number of one and another class, so the classes are not unevenly represented. 
  What happened when I train the model is that I am stuck at same loss while trening,...
  I assume that model predict always same class and half in batch has loss 0 half of them max, so that bring it to 8 all the time,...
  What could went wrong?
  The network is something like this : 
  x = Conv2D(16, (3, 3), padding='same')(input_img)
  x = LeakyReLU(0.1)(x)
  x = Conv2D(32 , (3, 3), padding='same')(x)
  x = LeakyReLU(0.1)(x)
  x = MaxPooling2D((2, 2))(x)
  x = Dropout(0.25)(x)
  x = Conv2D(32 , (3, 3), padding='same')(x)
  x = LeakyReLU(0.1)(x)
  x = Conv2D(48 , (3, 3), padding='same')(x)
  x = LeakyReLU(0.1)(x)
  x = MaxPooling2D((2, 2))(x)
  x = Dropout(0.25)(x)
  
  x = Flatten()(x)
  x = Dense(4096)(x)
  x = LeakyReLU(0.1)(x)
  x = Dropout(0.5)(x)
  x = Dense(2048)(x)
  x = LeakyReLU(0.1)(x)
  x = Dropout(0.5)(x)
  out = Dense(2, activation='softmax', name='table')(x)
  
  model = Model(input_img, out)
  model.compile(optimizer='adam', loss= 'categorical_crossentropy')
  
  Training Loss:
  
  **Tags:** convolutional-neural-networks, objective-functions, categorical-data
  > **Comments:**
  > **username_0**: I figure out, that sometimes bigger batch size helps : from 32 to 256,... But I would appreciate some theoretical and practical advice to solve this king of errors...  
  > **username_1**: There are many things that you could be doing wrong. 1. Are you sure your initialisation is correct? An unchanging error is sometimes a sign that an entire layer is dead, hence no ability to change the error. 2. Are you sure your backprop is updating everything? 3. Are you sure that you are definitely using a one hot vector as the output 4. If you try training your CNN on something else, is it able to learn? 5. Make sure your cross entropy loss function is correct. I can almost guarantee a behaviour like this is caused by an error in the final 2 or so layers of the network.  
  > **username_0**: @Recessive : 1. Initialisation is handle by Adam,.. and I use LeakyReLU not to produce 0 output 2. Yes 3. Yes I always use [0,1] or [1,0]. 4. I need to check this, but I think so (Usually if I increase batch size it train better, but I need to increase it to 256 or more,...).  5. I use default Keras loss function implementation. Regarding last layers : They are normal dense layers with LeakyReLU activations. What did you have in mind as a potential  error?  
  > **username_0**: I update my question, to give you some ideas how network look like,...  
  > **username_1**: Try taking the dropout out. What's the input dimensions of the image?  
  > **username_2**: Is this the training loss or validation loss ? Did you try to print out the predictions made by your network ?  
  > **username_0**: Even with dropouts, I get the same result. This is training loss (I update question). The model always predict only one class. This is the reason (I guess for constant loss, because half of the batch has zero loss, half max loss because it always outputs the same class).  
  
  ---
  
  ## 0 answers
  
  