Answers.Body: <p>Generally when you see metrics stagnate like this it's because the model has converged incorrectly (ex: always predicting 0, or gradients/weights have dropped to 0, etc.). Can you see if this is what's happening for your problem?</p>
  <p>I'd expect that perhaps your model has converged to predict the majority class for each label.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-27T02:17:15.567Z
Answers.LastActivityDate: 2021-09-27T02:17:15.567Z
Answers.UserDisplayName: Rohil Verma
Body: <p>I am rather new to deep learning and got some questions on performing a <strong>multi-label</strong> image classification task with keras convolutional neural networks. Those are mainly referring to evaluating keras models performing multi label classification tasks. I will structure this a bit to get a better overview first.</p>
  <h2>Problem Description</h2>
  <p>The underlying dataset are album cover images from different genres. In my case those are electronic, rock, jazz, pop, hiphop. So we have 5 possible classes that are not mutual exclusive. Task is to predict possible genres for a given album cover. Each album cover is of size 300px x 300px. The images are loaded into tensorflow datasets, resized to 150px x 150px.
  <a href="https://i.sstatic.net/k6ltO.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/k6ltO.png" alt="Dataset distribution in training, validation and test data." /></a></p>
  <h2>Model Architecture</h2>
  <p>The architecture for the model is the following.</p>
  <pre><code>import tensorflow as tf
  
  from tensorflow import keras
  from tensorflow.keras import layers
  from tensorflow.keras.models import Sequential
  
  data_augmentation = keras.Sequential(
    [
      layers.experimental.preprocessing.RandomFlip(&quot;horizontal&quot;, 
                                                   input_shape=(img_height, 
                                                                img_width,
                                                                3)),
     layers.experimental.preprocessing.RandomFlip(&quot;vertical&quot;),
      layers.experimental.preprocessing.RandomRotation(0.4),
     layers.experimental.preprocessing.RandomZoom(height_factor=(0.2, 0.6), width_factor=(0.2, 0.6))
    ]
  )
  
  def create_model(num_classes=5, augmentation_layers=None):
    model = Sequential()
  
    # We can pass a list of layers performing data augmentation here
    if augmentation_layers:
      # The first layer of the augmentation layers must define the input shape
      model.add(augmentation_layers)
      model.add(layers.experimental.preprocessing.Rescaling(1./255))
    else:
      model.add(layers.experimental.preprocessing.Rescaling(1./255, input_shape=(img_height, img_width, 3)))
  
    model.add(layers.Conv2D(32, (3, 3), activation='relu'))
    model.add(layers.MaxPooling2D((2, 2)))
    model.add(layers.Conv2D(64, (3, 3), activation='relu'))
    model.add(layers.MaxPooling2D((2, 2)))
    model.add(layers.Conv2D(128, (3, 3), activation='relu'))
    model.add(layers.MaxPooling2D((2, 2)))
    model.add(layers.Conv2D(128, (3, 3), activation='relu'))
    model.add(layers.MaxPooling2D((2, 2)))
    model.add(layers.Flatten())
    model.add(layers.Dense(512, activation='relu'))
  
    # Use sigmoid activation function. Basically we train binary classifiers for each class by specifiying binary crossentropy loss and sigmoid activation on the output layer.
    model.add(layers.Dense(num_classes, activation='sigmoid'))
    model.summary()
  
    return model
  </code></pre>
  <p>I'm not using the usual metrics here like standard accuracy. In this <a href="http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.364.5612&amp;rep=rep1&amp;type=pdf" rel="nofollow noreferrer">paper</a> I read that you cannot evaluate multi-label classification models with the usual methods. In chapter 7. evaluation metrics the hamming loss and an adjusted accuracy (variant of exact match) are presented which I use for this model.</p>
  <p>The hamming loss is already provided by tensorflow-addons (<a href="https://www.tensorflow.org/addons/api_docs/python/tfa/metrics/HammingLoss" rel="nofollow noreferrer">see here</a>) and an implementation of the subset accuracy I found here (<a href="https://stackoverflow.com/questions/56739708/how-to-implement-exact-match-subset-accuracy-as-a-metric-for-keras">see here</a>).</p>
  <pre><code>from tensorflow_addons.metrics import HammingLoss
  
  hamming_loss = HammingLoss(mode=&quot;multilabel&quot;, threshold=0.5)
  
  def subset_accuracy(y_true, y_pred):
      # From https://stackoverflow.com/questions/56739708/how-to-implement-exact-match-subset-accuracy-as-a-metric-for-keras
  
      threshold = tf.constant(.5, tf.float32)
      gtt_pred = tf.math.greater(y_pred, threshold)
      gtt_true = tf.math.greater(y_true, threshold)
      accuracy = tf.reduce_mean(tf.cast(tf.equal(gtt_pred, gtt_true), tf.float32), axis=-1)
      return accuracy
  
   # Create model
   model = create_model(num_classes=5, augmentation_layers=data_augmentation)
  
   # Compile model  
   model.compile(loss=&quot;binary_crossentropy&quot;, optimizer=&quot;adam&quot;, metrics=[subset_accuracy, hamming_loss])
  
   # Fit the model
   history = model.fit(training_dataset, epochs=epochs, validation_data=validation_dataset, callbacks=callbacks)
  
  </code></pre>
  <h2>Problem with this model</h2>
  <p>When training the model  subset_accuracy hamming_loss are at some point stuck which looks like the following:
  <a href="https://i.sstatic.net/Ry3Zl.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Ry3Zl.png" alt="Training History" /></a></p>
  <p>What could cause this behaviour? I am honestly a little bit lost right now. Could this be a case of the dying ReLU problem? Or is it wrong use of the metrics mentioned or is the implementation of those maybe wrong?</p>
  <p>So far, I tried to test different optimizers and lowering the learning rate (e.g. from 0.01 to 0.001, 0.0001, etc..) but that didn't help either.</p>
  
Tags: <tensorflow><keras><performance><multi-label-classification>
Title: What could cause the hamming loss and subset accuracy to get stuck in a multi-label image classification problem?
UserDisplayName: Phil
ThreadText: # What could cause the hamming loss and subset accuracy to get stuck in a multi-label image classification problem?
  
  (Asked by: username_0 on 2021-09-26)
  
  I am rather new to deep learning and got some questions on performing a **multi-label** image classification task with keras convolutional neural networks. Those are mainly referring to evaluating keras models performing multi label classification tasks. I will structure this a bit to get a better overview first.
  
  ## Problem Description
  
  The underlying dataset are album cover images from different genres. In my case those are electronic, rock, jazz, pop, hiphop. So we have 5 possible classes that are not mutual exclusive. Task is to predict possible genres for a given album cover. Each album cover is of size 300px x 300px. The images are loaded into tensorflow datasets, resized to 150px x 150px.
  [![Dataset distribution in training, validation and test data.](https://i.sstatic.net/k6ltO.png)](https://i.sstatic.net/k6ltO.png)
  
  ## Model Architecture
  
  The architecture for the model is the following.
  
  ```
  import tensorflow as tf
  
  from tensorflow import keras
  from tensorflow.keras import layers
  from tensorflow.keras.models import Sequential
  
  data_augmentation = keras.Sequential(
    [
      layers.experimental.preprocessing.RandomFlip("horizontal", 
                                                   input_shape=(img_height, 
                                                                img_width,
                                                                3)),
     layers.experimental.preprocessing.RandomFlip("vertical"),
      layers.experimental.preprocessing.RandomRotation(0.4),
     layers.experimental.preprocessing.RandomZoom(height_factor=(0.2, 0.6), width_factor=(0.2, 0.6))
    ]
  )
  
  def create_model(num_classes=5, augmentation_layers=None):
    model = Sequential()
  
    # We can pass a list of layers performing data augmentation here
    if augmentation_layers:
      # The first layer of the augmentation layers must define the input shape
      model.add(augmentation_layers)
      model.add(layers.experimental.preprocessing.Rescaling(1./255))
    else:
      model.add(layers.experimental.preprocessing.Rescaling(1./255, input_shape=(img_height, img_width, 3)))
  
    model.add(layers.Conv2D(32, (3, 3), activation='relu'))
    model.add(layers.MaxPooling2D((2, 2)))
    model.add(layers.Conv2D(64, (3, 3), activation='relu'))
    model.add(layers.MaxPooling2D((2, 2)))
    model.add(layers.Conv2D(128, (3, 3), activation='relu'))
    model.add(layers.MaxPooling2D((2, 2)))
    model.add(layers.Conv2D(128, (3, 3), activation='relu'))
    model.add(layers.MaxPooling2D((2, 2)))
    model.add(layers.Flatten())
    model.add(layers.Dense(512, activation='relu'))
  
    # Use sigmoid activation function. Basically we train binary classifiers for each class by specifiying binary crossentropy loss and sigmoid activation on the output layer.
    model.add(layers.Dense(num_classes, activation='sigmoid'))
    model.summary()
  
    return model
  
  ```
  
  I'm not using the usual metrics here like standard accuracy. In this [paper](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.364.5612&rep=rep1&type=pdf) I read that you cannot evaluate multi-label classification models with the usual methods. In chapter 7. evaluation metrics the hamming loss and an adjusted accuracy (variant of exact match) are presented which I use for this model.
  
  The hamming loss is already provided by tensorflow-addons ([see here](https://www.tensorflow.org/addons/api_docs/python/tfa/metrics/HammingLoss)) and an implementation of the subset accuracy I found here ([see here](https://stackoverflow.com/questions/56739708/how-to-implement-exact-match-subset-accuracy-as-a-metric-for-keras)).
  
  ```
  from tensorflow_addons.metrics import HammingLoss
  
  hamming_loss = HammingLoss(mode="multilabel", threshold=0.5)
  
  def subset_accuracy(y_true, y_pred):
      # From https://stackoverflow.com/questions/56739708/how-to-implement-exact-match-subset-accuracy-as-a-metric-for-keras
  
      threshold = tf.constant(.5, tf.float32)
      gtt_pred = tf.math.greater(y_pred, threshold)
      gtt_true = tf.math.greater(y_true, threshold)
      accuracy = tf.reduce_mean(tf.cast(tf.equal(gtt_pred, gtt_true), tf.float32), axis=-1)
      return accuracy
  
   # Create model
   model = create_model(num_classes=5, augmentation_layers=data_augmentation)
  
   # Compile model  
   model.compile(loss="binary_crossentropy", optimizer="adam", metrics=[subset_accuracy, hamming_loss])
  
   # Fit the model
   history = model.fit(training_dataset, epochs=epochs, validation_data=validation_dataset, callbacks=callbacks)
  
  ```
  
  ## Problem with this model
  
  When training the model subset\_accuracy hamming\_loss are at some point stuck which looks like the following:
  [![Training History](https://i.sstatic.net/Ry3Zl.png)](https://i.sstatic.net/Ry3Zl.png)
  
  What could cause this behaviour? I am honestly a little bit lost right now. Could this be a case of the dying ReLU problem? Or is it wrong use of the metrics mentioned or is the implementation of those maybe wrong?
  
  So far, I tried to test different optimizers and lowering the learning rate (e.g. from 0.01 to 0.001, 0.0001, etc..) but that didn't help either.
  
  **Tags:** tensorflow, keras, performance, multi-label-classification
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-09-27. Score: 0
  
  Generally when you see metrics stagnate like this it's because the model has converged incorrectly (ex: always predicting 0, or gradients/weights have dropped to 0, etc.). Can you see if this is what's happening for your problem?
  
  I'd expect that perhaps your model has converged to predict the majority class for each label.
  
  
  ---
  
  