Answers.Body: <p>I <em>think</em> you may have a <a href="http://www.chioka.in/class-imbalance-problem/" rel="nofollow noreferrer">class imbalance problem</a> here, if I am reading your output correctly. You have 20,000 negative examples, but only 8000 positive ones, and you are minimizing binary cross entropy without re-weighting the examples, so your model can achieve a low-ish loss just by consistently outputing a value close to 0. This forms a local optima in the search space for the model.</p>
  
  <p>To fix this, you could try to optimize some other loss function that is more sensitive to class imbalances, or, likely more productively, you could just use an equal number of examples for each class. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-19T03:20:21.563Z
Answers.Text: I've seen examples (cat vs non-cat) where they used 5K for non-cats and 1600 for cats. But what combo of activation/loss shall I use? Is it OK to change the `sigmoid` to `softmax` and still have the `binary_crossentropy`? (like the cat exmaple).
Answers.UserDisplayName: Mary
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-19T03:27:53.877Z
Answers.Text: @TinaJ whether you can handle class imbalance will really depend on the problem you are working on. If the classes are difficult to separate, then it will be a problem. If not, it probably won't be. I think the choice of activation function shouldn't make a big difference here.
Answers.UserDisplayName: John Doucette
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-19T03:31:35.277Z
Answers.Text: My impression was the more non-object I have, the better the accuracy. Didn't know the size imbalance can also be a negative factor!
Answers.UserDisplayName: Mary
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-19T03:34:02.130Z
Answers.Text: @TinaJ an extreme case that might help with the intuition: Imagine we have a dataset of blood test results, where 99.99% of the results are negative for a disease, and 0.01% are positive. A model can achieve an extremely low cross-entropy loss value by claiming everything is negative. It will be right almost always, but the model is not a useful one.
Answers.UserDisplayName: John Doucette
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-19T03:09:55.233Z
Answers.LastActivityDate: 2019-11-19T03:09:55.233Z
Answers.UserDisplayName: John Doucette
Body: <p>I'm new to Deep Learning. I used <code>Keras</code> and trained a <code>inception_resnet_v2</code> model for my binary classification application (fire detection). As suggested from my <a href="https://ai.stackexchange.com/questions/16268">previous question</a> of a <code>non-X</code> class, I prepared a dataset of 8000 images of fire, and a larger dataset for non-fire (20,000 random images) to make sure the network also sees images of non-fire to perform classification.</p>
  
  <p>I trained the model, but now when trying to load the model and pass images of fire and non-fire ones, it shows same result for all of them:</p>
  
  <pre><code>[[0. 1.]]
  [[0. 1.]]
  [[0. 1.]]
  [[0. 1.]]
  [[0. 1.]]
  </code></pre>
  
  <p>What is going wrong? Am I doing anything wrong? Should I get the result another way?</p>
  
  <p>===============================================</p>
  
  <p>I know it's not SO, but this is my prediction code in case it matters:</p>
  
  <pre><code>from __future__ import print_function
  from keras.models import load_model, model_from_json
  import cv2, os, glob
  import numpy as np
  from keras.preprocessing import image
  
  if __name__ == '__main__':
      model = load_model('Resnet_26_0.79_model_weights.h5')
  
      os.chdir("test")
      for file in glob.glob("*.jpg"):
          img_path = file
          img = image.load_img(img_path, target_size=(300, 300))
          x = image.img_to_array(img)
          x = np.expand_dims(x, axis=0)
  
          dictionary = {0: 'non-fire', 1: 'fire'}
  
          results = model.predict(x)
          print(results)
          predicted_class= np.argmax(results)
          acc = 100*results[0][predicted_class]
          print("Network prediction is: file: "+ file+", "+dictionary[predicted_class]+", %{:0.2f}".format(acc))
  </code></pre>
  
  <p>And here is the training:</p>
  
  <pre><code>from keras.applications.inception_resnet_v2 import InceptionResNetV2, preprocess_input
  from keras.preprocessing.image import ImageDataGenerator
  from keras.layers import Dense, Activation, Flatten, Dropout
  from keras.models import Sequential, Model
  from keras.optimizers import SGD, Adam
  from keras.callbacks import ModelCheckpoint
  from keras.metrics import binary_accuracy
  import os
  import json
  #==========================
  HEIGHT = 300
  WIDTH = 300
  TRAIN_DIR = "data"
  BATCH_SIZE = 8 #8
  steps_per_epoch = 1000 #1000
  NUM_EPOCHS = 50 #50
  lr= 0.00001
  #==========================
  FC_LAYERS = [1024, 1024]
  dropout = 0.5
  
  def build_finetune_model(base_model, dropout, fc_layers, num_classes):
      for layer in base_model.layers:
          layer.trainable = False
  
      x = base_model.output
      x = Flatten()(x)
      for fc in fc_layers:
          # New FC layer, random init
          x = Dense(fc, activation='relu')(x) 
          x = Dropout(dropout)(x)
  
      # New layer
      predictions = Dense(num_classes, activation='sigmoid')(x) 
      finetune_model = Model(inputs=base_model.input, outputs=predictions)
      return finetune_model
  
  train_datagen =  ImageDataGenerator(preprocessing_function=preprocess_input, rotation_range=90, horizontal_flip=True, vertical_flip=True
                                      ,validation_split=0.2)
  train_generator = train_datagen.flow_from_directory(TRAIN_DIR, target_size=(HEIGHT, WIDTH), batch_size=BATCH_SIZE
                                                      ,subset="training")
  #split validation manually
  validation_generator = train_datagen.flow_from_directory(TRAIN_DIR, target_size=(HEIGHT, WIDTH), batch_size=BATCH_SIZE,subset="validation")
  
  base_model = InceptionResNetV2(weights='imagenet', include_top=False, input_shape=(HEIGHT, WIDTH, 3))
  
  root=TRAIN_DIR
  class_list = [ item for item in os.listdir(root) if os.path.isdir(os.path.join(root, item)) ]
  print ("class_list: "+str(class_list))
  
  finetune_model = build_finetune_model(base_model, dropout=dropout, fc_layers=FC_LAYERS, num_classes=len(class_list))
  
  adam = Adam(lr)
  # change to categorical_crossentropy for multiple classes
  finetune_model.compile(adam, loss='binary_crossentropy', metrics=['accuracy'])
  
  filepath="./checkpoints/" + "Resnet_{epoch:02d}_{acc:.2f}" +"_model_weights.h5"
  checkpoint = ModelCheckpoint(filepath, monitor=["val_accuracy"], verbose=1, mode='max', save_weights_only=False)
  callbacks_list = [checkpoint]
  
  history = finetune_model.fit_generator(train_generator, epochs=NUM_EPOCHS, workers=BATCH_SIZE, 
                                      validation_data=validation_generator, validation_steps = validation_generator.samples, 
                                         steps_per_epoch=steps_per_epoch, 
                                         shuffle=True, callbacks=callbacks_list)
  </code></pre>
  
Tags: <deep-learning><convolutional-neural-networks><classification><training><keras>
Title: Semantic issues with predictions made by my trained model
UserDisplayName: Mary
ThreadText: # Semantic issues with predictions made by my trained model
  
  (Asked by: username_0 on 2019-11-18)
  
  I'm new to Deep Learning. I used `Keras` and trained a `inception_resnet_v2` model for my binary classification application (fire detection). As suggested from my [previous question](https://ai.stackexchange.com/questions/16268) of a `non-X` class, I prepared a dataset of 8000 images of fire, and a larger dataset for non-fire (20,000 random images) to make sure the network also sees images of non-fire to perform classification.
  
  I trained the model, but now when trying to load the model and pass images of fire and non-fire ones, it shows same result for all of them:
  
  ```
  [[0. 1.]]
  [[0. 1.]]
  [[0. 1.]]
  [[0. 1.]]
  [[0. 1.]]
  
  ```
  
  What is going wrong? Am I doing anything wrong? Should I get the result another way?
  
  ===============================================
  
  I know it's not SO, but this is my prediction code in case it matters:
  
  ```
  from __future__ import print_function
  from keras.models import load_model, model_from_json
  import cv2, os, glob
  import numpy as np
  from keras.preprocessing import image
  
  if __name__ == '__main__':
      model = load_model('Resnet_26_0.79_model_weights.h5')
  
      os.chdir("test")
      for file in glob.glob("*.jpg"):
          img_path = file
          img = image.load_img(img_path, target_size=(300, 300))
          x = image.img_to_array(img)
          x = np.expand_dims(x, axis=0)
  
          dictionary = {0: 'non-fire', 1: 'fire'}
  
          results = model.predict(x)
          print(results)
          predicted_class= np.argmax(results)
          acc = 100*results[0][predicted_class]
          print("Network prediction is: file: "+ file+", "+dictionary[predicted_class]+", %{:0.2f}".format(acc))
  
  ```
  
  And here is the training:
  
  ```
  from keras.applications.inception_resnet_v2 import InceptionResNetV2, preprocess_input
  from keras.preprocessing.image import ImageDataGenerator
  from keras.layers import Dense, Activation, Flatten, Dropout
  from keras.models import Sequential, Model
  from keras.optimizers import SGD, Adam
  from keras.callbacks import ModelCheckpoint
  from keras.metrics import binary_accuracy
  import os
  import json
  #==========================
  HEIGHT = 300
  WIDTH = 300
  TRAIN_DIR = "data"
  BATCH_SIZE = 8 #8
  steps_per_epoch = 1000 #1000
  NUM_EPOCHS = 50 #50
  lr= 0.00001
  #==========================
  FC_LAYERS = [1024, 1024]
  dropout = 0.5
  
  def build_finetune_model(base_model, dropout, fc_layers, num_classes):
      for layer in base_model.layers:
          layer.trainable = False
  
      x = base_model.output
      x = Flatten()(x)
      for fc in fc_layers:
          # New FC layer, random init
          x = Dense(fc, activation='relu')(x) 
          x = Dropout(dropout)(x)
  
      # New layer
      predictions = Dense(num_classes, activation='sigmoid')(x) 
      finetune_model = Model(inputs=base_model.input, outputs=predictions)
      return finetune_model
  
  train_datagen =  ImageDataGenerator(preprocessing_function=preprocess_input, rotation_range=90, horizontal_flip=True, vertical_flip=True
                                      ,validation_split=0.2)
  train_generator = train_datagen.flow_from_directory(TRAIN_DIR, target_size=(HEIGHT, WIDTH), batch_size=BATCH_SIZE
                                                      ,subset="training")
  #split validation manually
  validation_generator = train_datagen.flow_from_directory(TRAIN_DIR, target_size=(HEIGHT, WIDTH), batch_size=BATCH_SIZE,subset="validation")
  
  base_model = InceptionResNetV2(weights='imagenet', include_top=False, input_shape=(HEIGHT, WIDTH, 3))
  
  root=TRAIN_DIR
  class_list = [ item for item in os.listdir(root) if os.path.isdir(os.path.join(root, item)) ]
  print ("class_list: "+str(class_list))
  
  finetune_model = build_finetune_model(base_model, dropout=dropout, fc_layers=FC_LAYERS, num_classes=len(class_list))
  
  adam = Adam(lr)
  # change to categorical_crossentropy for multiple classes
  finetune_model.compile(adam, loss='binary_crossentropy', metrics=['accuracy'])
  
  filepath="./checkpoints/" + "Resnet_{epoch:02d}_{acc:.2f}" +"_model_weights.h5"
  checkpoint = ModelCheckpoint(filepath, monitor=["val_accuracy"], verbose=1, mode='max', save_weights_only=False)
  callbacks_list = [checkpoint]
  
  history = finetune_model.fit_generator(train_generator, epochs=NUM_EPOCHS, workers=BATCH_SIZE, 
                                      validation_data=validation_generator, validation_steps = validation_generator.samples, 
                                         steps_per_epoch=steps_per_epoch, 
                                         shuffle=True, callbacks=callbacks_list)
  
  ```
  
  **Tags:** deep-learning, convolutional-neural-networks, classification, training, keras
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-11-19. Score: 1
  
  I *think* you may have a [class imbalance problem](http://www.chioka.in/class-imbalance-problem/) here, if I am reading your output correctly. You have 20,000 negative examples, but only 8000 positive ones, and you are minimizing binary cross entropy without re-weighting the examples, so your model can achieve a low-ish loss just by consistently outputing a value close to 0. This forms a local optima in the search space for the model.
  
  To fix this, you could try to optimize some other loss function that is more sensitive to class imbalances, or, likely more productively, you could just use an equal number of examples for each class.
  
  > **Comments:**
  > **username_0**: I've seen examples (cat vs non-cat) where they used 5K for non-cats and 1600 for cats. But what combo of activation/loss shall I use? Is it OK to change the `sigmoid` to `softmax` and still have the `binary_crossentropy`? (like the cat exmaple).  
  > **username_1**: @TinaJ whether you can handle class imbalance will really depend on the problem you are working on. If the classes are difficult to separate, then it will be a problem. If not, it probably won't be. I think the choice of activation function shouldn't make a big difference here.  
  > **username_0**: My impression was the more non-object I have, the better the accuracy. Didn't know the size imbalance can also be a negative factor!  
  > **username_1**: @TinaJ an extreme case that might help with the intuition: Imagine we have a dataset of blood test results, where 99.99% of the results are negative for a disease, and 0.01% are positive. A model can achieve an extremely low cross-entropy loss value by claiming everything is negative. It will be right almost always, but the model is not a useful one.  
  
  ---
  
  