Body: <p>I have run into a strange behavior of my <strong>multi label</strong> classification ANN</p>
  <pre><code>model = Sequential()
  model.add(Dense(6, input_shape=(input_size,), activation='elu'))
  #model.add(BatchNormalization(axis=-1))
  model.add(Dropout(0.2))
  #model.add(BatchNormalization(axis=-1))
  model.add(Dense(6, activation='elu'))
  model.add(Dropout(0.2))
  #model.add(BatchNormalization(axis=-1))
  model.add(Dense(6, activation='elu'))
  model.add(Dropout(0.2))
  
  # model.add(keras.layers.BatchNormalization(axis=-1))
  model.add(Dense(6, activation='sigmoid'))
  model.compile(loss='binary_crossentropy',
                optimizer='nadam',
                metrics=['accuracy'])
  history = model.fit(X_train, Y_train,batch_size=64 ,epochs=300,
                      validation_data = (X_test, Y_test), verbose=2)
  </code></pre>
  <p>The result is quite strange, I have a feeling that my model could not improve any more. Why does the loss and the accuracy does not change overtime ?</p>
  <p>P/S For clarification, I have 6 output and the value of each output is 0 or 1 that is
  output 1: can be 0 or 1</p>
  <p>output 2: can be 0 or 1</p>
  <p>output 3: can be 0 or 1</p>
  <p>output 4: can be 0 or 1</p>
  <p>output 5: can be 0 or 1</p>
  <p>output 6: can be 0 or 1</p>
  <p><a href="https://i.sstatic.net/1rUqs.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/1rUqs.png" alt="enter image description here" /></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-18T08:28:02.397Z
Comments.Text: Hi ! How many samples do you have in X_train ? Can you try to visualize the outputs on your test set (for instance, are your predictions constants ?) ? Maybe one problem is that you put a lot of Dropout, and your network is already narrow so instead of preventing overfitting this could hinder the learning...
Comments.UserDisplayName: 16Aghnar
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-18T10:40:15.523Z
Comments.Text: about 4000 training samples, and validate 1000 samples.The output for each test set data point is a binary output vector that is 
  
  [out1 out2 out3 out4 out5 out6]
  
  
  Particularly
  
   output 1: can be 0 or 1
  
  output 2: can be 0 or 1
  
  output 3: can be 0 or 1
  
  output 4: can be 0 or 1
  
  output 5: can be 0 or 1
  
  output 6: can be 0 or 1
Comments.UserDisplayName: User2741
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-18T11:30:11.323Z
Comments.Text: Okay. Why did you use Dropout and Batch norm ? It may be useful to visualize the actual predictions of your trained net. You know, using for instance `plt.scatter(Y_test, model.predict(X_test))`
Comments.UserDisplayName: 16Aghnar
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-18T11:37:13.747Z
Comments.Text: (As you have multi label classification, you may plot `plt.scatter(Y_test, model.predict(X_test))` for each particular label)
Comments.UserDisplayName: 16Aghnar
Tags: <neural-networks><accuracy><loss>
Title: Why does loss and accuracy for a multi label classification ann does not change overtime?
UserDisplayName: User2741
ThreadText: # Why does loss and accuracy for a multi label classification ann does not change overtime?
  
  (Asked by: username_0 on 2020-08-18)
  
  I have run into a strange behavior of my multi label classification ANN
  model = Sequential()
  model.add(Dense(6, input_shape=(input_size,), activation='elu'))
  #model.add(BatchNormalization(axis=-1))
  model.add(Dropout(0.2))
  #model.add(BatchNormalization(axis=-1))
  model.add(Dense(6, activation='elu'))
  model.add(Dropout(0.2))
  #model.add(BatchNormalization(axis=-1))
  model.add(Dense(6, activation='elu'))
  model.add(Dropout(0.2))
  
  # model.add(keras.layers.BatchNormalization(axis=-1))
  model.add(Dense(6, activation='sigmoid'))
  model.compile(loss='binary_crossentropy',
                optimizer='nadam',
                metrics=['accuracy'])
  history = model.fit(X_train, Y_train,batch_size=64 ,epochs=300,
                      validation_data = (X_test, Y_test), verbose=2)
  
  The result is quite strange, I have a feeling that my model could not improve any more. Why does the loss and the accuracy does not change overtime ?
  P/S For clarification, I have 6 output and the value of each output is 0 or 1 that is
  output 1: can be 0 or 1
  output 2: can be 0 or 1
  output 3: can be 0 or 1
  output 4: can be 0 or 1
  output 5: can be 0 or 1
  output 6: can be 0 or 1
  
  **Tags:** neural-networks, accuracy, loss
  > **Comments:**
  > **username_1**: Hi ! How many samples do you have in X_train ? Can you try to visualize the outputs on your test set (for instance, are your predictions constants ?) ? Maybe one problem is that you put a lot of Dropout, and your network is already narrow so instead of preventing overfitting this could hinder the learning...  
  > **username_0**: about 4000 training samples, and validate 1000 samples.The output for each test set data point is a binary output vector that is   [out1 out2 out3 out4 out5 out6]   Particularly   output 1: can be 0 or 1  output 2: can be 0 or 1  output 3: can be 0 or 1  output 4: can be 0 or 1  output 5: can be 0 or 1  output 6: can be 0 or 1  
  > **username_1**: Okay. Why did you use Dropout and Batch norm ? It may be useful to visualize the actual predictions of your trained net. You know, using for instance `plt.scatter(Y_test, model.predict(X_test))`  
  > **username_1**: (As you have multi label classification, you may plot `plt.scatter(Y_test, model.predict(X_test))` for each particular label)  
  
  ---
  
  ## 0 answers
  
  