Answers.Body: <p>The paper <a href="https://arxiv.org/pdf/1710.05381.pdf" rel="nofollow noreferrer">A systematic study of the class imbalance problem
  in convolutional neural networks</a> is a great overview on class imbalance approaches. Section 2 summarizes various methods commonly used. They categorize &quot;Adding Class Weights for an imbalanced dataset&quot; under the technique &quot;Cost sensitive learning&quot;:</p>
  <blockquote>
  <p><strong>Cost sensitive learning.</strong> This method assigns different cost to misclassification of examples from different classes [44]. With respect to neural networks it can be implemented in various ways. One approach is threshold moving [22] or post scaling [23] that is applied in the inference phase after the classifier is already trained. Similar strategy is to adapt the output of the network and also use it in the backward pass of backpropagation algorithm [45]. Another adaptation of neural network to be cost sensitive is to modify the learning rate such that higher cost examples contribute more to the update of weights. And finally we can train the network by minimizing the misclassification cost instead of standard loss function</p>
  </blockquote>
  <p>Without further context, &quot;Adding Class Weights for an imbalanced dataset&quot; can mean many things as enumerated by the above. But if I had to guess, the most common meaning is that they weigh the misclassification cost differently per label by multiplying it with a different weight variable.</p>
  <p>For example, maybe label 1, 2, and 3, when misclassified, gets a 1x multiplier in the training loss (the standard weight), but label 4 gets a 3x multiplier because it is roughly 3x more important, and label 5 gets a 0.5x multiplier because it is 0.5x less important.</p>
  <p><a href="https://datascience.stackexchange.com/questions/13490/how-to-set-class-weights-for-imbalanced-classes-in-keras">Here</a> is an example of how to do this with Keras.</p>
  <p>(FYI the paper above recommends you over-sample the minority class rather than using the cost-sensitive learning approach to help with class imbalance)</p>
  <p>As for missing labels, the quickest solution is just to skip those training instances when training for that particular missing label. If you want to still use the training instances somehow, a common approach is to fix the missing label with <a href="http://www.cs.cmu.edu/%7E10701/slides/17_SSL.pdf" rel="nofollow noreferrer">semi-supervised learning approaches</a>.</p>
  <p>One way to do semi-supervised learning is <strong>self-training</strong>, where your model is trained on only the labeled instances, and then makes predictions on the missing labels. High-confidence predictions of the missing labels are then added to the training data, and the model is trained on the new training data. This process repeats until convergence of the training data.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-31T18:58:29.630Z
Answers.LastActivityDate: 2020-12-31T18:58:29.630Z
Answers.UserDisplayName: user3667125
Body: <p>Suppose I have the following toy data set:</p>
  <p><a href="https://i.sstatic.net/yIuLn.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/yIuLn.png" alt="enter image description here" /></a></p>
  <p>Each instance has multiple labels at a time.</p>
  <p>You can see I have 2 instances for <strong>Label2</strong>. However, only one instance for the other labels. It means that we have class imbalanced issues.</p>
  <p>I read about adding class weights for an imbalanced dataset. However, I could not understand how it actually works and why it is beneficial.</p>
  <p>Can anyone explain this method generally, as well as according to my given toy data set?</p>
  <p>In addition to that, how do we handle these missing labels (nan)?</p>
  
Tags: <machine-learning><weights><multi-label-classification><imbalanced-datasets>
Title: What does "adding class weights for an imbalanced dataset" mean in the case of multi-label classification?
UserDisplayName: Nafees Ahmed
ThreadText: # What does "adding class weights for an imbalanced dataset" mean in the case of multi-label classification?
  
  (Asked by: username_0 on 2020-12-30)
  
  Suppose I have the following toy data set:
  
  [![enter image description here](https://i.sstatic.net/yIuLn.png)](https://i.sstatic.net/yIuLn.png)
  
  Each instance has multiple labels at a time.
  
  You can see I have 2 instances for **Label2**. However, only one instance for the other labels. It means that we have class imbalanced issues.
  
  I read about adding class weights for an imbalanced dataset. However, I could not understand how it actually works and why it is beneficial.
  
  Can anyone explain this method generally, as well as according to my given toy data set?
  
  In addition to that, how do we handle these missing labels (nan)?
  
  **Tags:** machine-learning, weights, multi-label-classification, imbalanced-datasets
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-12-31. Score: 1
  
  The paper [A systematic study of the class imbalance problem
  in convolutional neural networks](https://arxiv.org/pdf/1710.05381.pdf) is a great overview on class imbalance approaches. Section 2 summarizes various methods commonly used. They categorize "Adding Class Weights for an imbalanced dataset" under the technique "Cost sensitive learning":
  
  > **Cost sensitive learning.** This method assigns different cost to misclassification of examples from different classes [44]. With respect to neural networks it can be implemented in various ways. One approach is threshold moving [22] or post scaling [23] that is applied in the inference phase after the classifier is already trained. Similar strategy is to adapt the output of the network and also use it in the backward pass of backpropagation algorithm [45]. Another adaptation of neural network to be cost sensitive is to modify the learning rate such that higher cost examples contribute more to the update of weights. And finally we can train the network by minimizing the misclassification cost instead of standard loss function
  
  Without further context, "Adding Class Weights for an imbalanced dataset" can mean many things as enumerated by the above. But if I had to guess, the most common meaning is that they weigh the misclassification cost differently per label by multiplying it with a different weight variable.
  
  For example, maybe label 1, 2, and 3, when misclassified, gets a 1x multiplier in the training loss (the standard weight), but label 4 gets a 3x multiplier because it is roughly 3x more important, and label 5 gets a 0.5x multiplier because it is 0.5x less important.
  
  [Here](https://datascience.stackexchange.com/questions/13490/how-to-set-class-weights-for-imbalanced-classes-in-keras) is an example of how to do this with Keras.
  
  (FYI the paper above recommends you over-sample the minority class rather than using the cost-sensitive learning approach to help with class imbalance)
  
  As for missing labels, the quickest solution is just to skip those training instances when training for that particular missing label. If you want to still use the training instances somehow, a common approach is to fix the missing label with [semi-supervised learning approaches](http://www.cs.cmu.edu/%7E10701/slides/17_SSL.pdf).
  
  One way to do semi-supervised learning is **self-training**, where your model is trained on only the labeled instances, and then makes predictions on the missing labels. High-confidence predictions of the missing labels are then added to the training data, and the model is trained on the new training data. This process repeats until convergence of the training data.
  
  
  ---
  
  