Answers.Body: <p>Since you have a multiclass classification problem rather than a binary classification problem (i.e. a two-class problem), I recommend to adjust your architecture and use <code>softmax</code> instead of <code>sigmoid</code> as final activation function and <code>categorcal_crossentropy</code> instead of <code>binary_crossentropy</code>.</p>
  <p>Softmax will ensure all your outputs are valid probabilities. This is crucial in calculating the loss of your model. Using sigmoid will only ensure this if you have binary classes. Further categorical crossentropy is used for multiclass problems, again this is to make sure you calculate the correct loss. This is just a general model characteristic:<br />
  <em>binary</em> -&gt; <code>sigmoid</code> &amp; <code>binary_crossentropy</code><br />
  <em>multi-class</em> -&gt; <code>softmax</code> &amp; <code>(sparse_)categorical_crossentropy</code></p>
  <p>To answer you original question, I think you should use <code>average = micro</code>, <code>macro</code> or <code>weighted</code> when calculating the F1-score. This depends on what you like your model to put more emphasis on and how severe your class imbalances are.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-11T13:41:10.920Z
Answers.Text: Tinu thank you for the answer. Please check my update for the reasoning behind the use of sigmoid and binary crossentropy :)
Answers.UserDisplayName: NikSp
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-09T07:15:55.410Z
Answers.Text: This answer is incorrect. `Softmax` with `categorical cross-entropy` will result in a single distribution, which is needed for a multi-class, single-label output. OP is trying to work with a multi-label multi-class output and therefore, each class has to be predicted independently as a probability between 0 to 1. This requires a `sigmoid` with `binary_crossentropy`
Answers.UserDisplayName: Akshay Sehgal
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-11T13:21:01.130Z
Answers.LastActivityDate: 2020-07-11T13:21:01.130Z
Answers.UserDisplayName: Tinu
Body: <p>I would like to ask a question about the relationship of accuracy with the loss function.</p>
  <p>My experiment is a multiclass text classification problem, and I have built a Keras neural network to tackle it. My labels are something like</p>
  <p>array([array([0, 0, 0, 0, 0, 1, 0, 1]), array([0, 1, 1, 0, 0, 0, 0, 1])])</p>
  <p>For the final output layer I use the 'sigmoid' activation function and for loss the 'binary crossentropy', however, I am a bit confused about the metric. I am using the F1_score metric because Accuracy it's not a metric to count on when there are many more negative labels than positive labels. So, since the problem is multilabel classification shall I use the multi-label mode like tfa.metrics.F1_score(mode=&quot;micro&quot;). Is that correct? or since I use binary_crossentropy and sigmoid activation function should I use the standard binary f1-score because every label-tag Is independent to the others and has a different Bernoulli distribution?</p>
  <p>I would really like to get your input on this. My humble opinion is that I should you the binary standard mode of binary f1-score and not the multi-label micro approach even though my experiment is multi-label text classification.</p>
  <p><strong>My current approach (using micro F1-score since my y_train is Multi-label</strong></p>
  <pre><code>model_for_pruning.compile(optimizer='adam',
                            loss='binary_crossentropy',
                            metrics=[tfa.metrics.F1Score(y_train[0].shape[-1], average=&quot;micro&quot;)])
  </code></pre>
  <p><strong>My alternative approach (based on the binary_crossentropy and the sigmoid activation function, despite I have multi-label y_train)</strong></p>
  <pre><code>model_for_pruning.compile(optimizer='adam',
                            loss='binary_crossentropy',
                            metrics=[tfa.metrics.F1Score(y_train[0].shape[-1], average=None)])
  </code></pre>
  <p>Τhe reason why I use sigmoid and not softmax as the output layer</p>
  <p><a href="https://medium.com/towards-artificial-intelligence/keras-for-multi-label-text-classification-86d194311d0e" rel="nofollow noreferrer">relevant link</a>
  Why Sigmoid and not Softmax in the final dense layer?
  In the final layer of the above architecture, sigmoid function as been used instead of softmax. The advantage of using sigmoid over Softmax lies in the fact that one synopsis may have many possible genres. Using the Softmax function would imply that the probability of occurrence of one genre depends on the occurrence of other genres. But for this application, we need a function that would give scores for the occurrence of genres, which would be independent of occurrences of any other movie genre.</p>
  <p><a href="https://www.pyimagesearch.com/2018/05/07/multi-label-classification-with-keras/" rel="nofollow noreferrer">relevant link 2</a>
  Binary cross-entropy rather than categorical cross-entropy.
  This may seem counterintuitive for multi-label classification; however, the goal is to treat each output label as an independent Bernoulli distribution and we want to penalize each output node independently.</p>
  <p>Please check my reasoning behind this and I would be happy if we can contradict this explanation. To better explain my experiment, I want to predict movie genres. So a movie can belong to 1 or more genres ['Action', 'Comedy', 'Children'], so when I use softmax the probability sum to 1, while when I use sigmoid its single probability of a class has a range between (0,1). Thus, if the predictions are correct the genres with the highest probabilities are those assigned to the movie. So imagine that my vector of prediction probabilities are something like [0.15, 0.12, 0.54, 0.78, 0.99] sum()&gt; 1, and not something like [0.12, 0.43, 11, 0.32,  0.01, 0.01) sum() = 1.</p>
  
Tags: <machine-learning><python><keras>
Title: Binary mode or Multi-label mode is correct when using binary crossentropy and sigmoid output function on multi-label classification
UserDisplayName: NikSp
ThreadText: # Binary mode or Multi-label mode is correct when using binary crossentropy and sigmoid output function on multi-label classification
  
  (Asked by: username_0 on 2020-07-10)
  
  I would like to ask a question about the relationship of accuracy with the loss function.
  My experiment is a multiclass text classification problem, and I have built a Keras neural network to tackle it. My labels are something like
  array([array([0, 0, 0, 0, 0, 1, 0, 1]), array([0, 1, 1, 0, 0, 0, 0, 1])])
  For the final output layer I use the 'sigmoid' activation function and for loss the 'binary crossentropy', however, I am a bit confused about the metric. I am using the F1_score metric because Accuracy it's not a metric to count on when there are many more negative labels than positive labels. So, since the problem is multilabel classification shall I use the multi-label mode like tfa.metrics.F1_score(mode="micro"). Is that correct? or since I use binary_crossentropy and sigmoid activation function should I use the standard binary f1-score because every label-tag Is independent to the others and has a different Bernoulli distribution?
  I would really like to get your input on this. My humble opinion is that I should you the binary standard mode of binary f1-score and not the multi-label micro approach even though my experiment is multi-label text classification.
  My current approach (using micro F1-score since my y_train is Multi-label
  model_for_pruning.compile(optimizer='adam',
                            loss='binary_crossentropy',
                            metrics=[tfa.metrics.F1Score(y_train[0].shape[-1], average="micro")])
  
  My alternative approach (based on the binary_crossentropy and the sigmoid activation function, despite I have multi-label y_train)
  model_for_pruning.compile(optimizer='adam',
                            loss='binary_crossentropy',
                            metrics=[tfa.metrics.F1Score(y_train[0].shape[-1], average=None)])
  
  Τhe reason why I use sigmoid and not softmax as the output layer
  relevant link
  Why Sigmoid and not Softmax in the final dense layer?
  In the final layer of the above architecture, sigmoid function as been used instead of softmax. The advantage of using sigmoid over Softmax lies in the fact that one synopsis may have many possible genres. Using the Softmax function would imply that the probability of occurrence of one genre depends on the occurrence of other genres. But for this application, we need a function that would give scores for the occurrence of genres, which would be independent of occurrences of any other movie genre.
  relevant link 2
  Binary cross-entropy rather than categorical cross-entropy.
  This may seem counterintuitive for multi-label classification; however, the goal is to treat each output label as an independent Bernoulli distribution and we want to penalize each output node independently.
  Please check my reasoning behind this and I would be happy if we can contradict this explanation. To better explain my experiment, I want to predict movie genres. So a movie can belong to 1 or more genres ['Action', 'Comedy', 'Children'], so when I use softmax the probability sum to 1, while when I use sigmoid its single probability of a class has a range between (0,1). Thus, if the predictions are correct the genres with the highest probabilities are those assigned to the movie. So imagine that my vector of prediction probabilities are something like [0.15, 0.12, 0.54, 0.78, 0.99] sum()> 1, and not something like [0.12, 0.43, 11, 0.32,  0.01, 0.01) sum() = 1.
  
  **Tags:** machine-learning, python, keras
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-07-11. Score: -1
  
  Since you have a multiclass classification problem rather than a binary classification problem (i.e. a two-class problem), I recommend to adjust your architecture and use `softmax` instead of `sigmoid` as final activation function and `categorcal_crossentropy` instead of `binary_crossentropy`.
  
  Softmax will ensure all your outputs are valid probabilities. This is crucial in calculating the loss of your model. Using sigmoid will only ensure this if you have binary classes. Further categorical crossentropy is used for multiclass problems, again this is to make sure you calculate the correct loss. This is just a general model characteristic:  
  *binary* -> `sigmoid` & `binary_crossentropy`  
  *multi-class* -> `softmax` & `(sparse_)categorical_crossentropy`
  
  To answer you original question, I think you should use `average = micro`, `macro` or `weighted` when calculating the F1-score. This depends on what you like your model to put more emphasis on and how severe your class imbalances are.
  
  > **Comments:**
  > **username_0**: Tinu thank you for the answer. Please check my update for the reasoning behind the use of sigmoid and binary crossentropy :)  
  > **username_2**: This answer is incorrect. `Softmax` with `categorical cross-entropy` will result in a single distribution, which is needed for a multi-class, single-label output. OP is trying to work with a multi-label multi-class output and therefore, each class has to be predicted independently as a probability between 0 to 1. This requires a `sigmoid` with `binary_crossentropy`  
  
  ---
  
  