Answers.Body: <p>In principle nothing looks wrong here. When using weighted cross entropy, the loss is adjusted based on frequency with a weight between 0 and 1. Now normally this works just fine, but since you have 500 classes I suspect your minority classes are boosted a lot. You can actually test it pretty easily.</p>
  <pre><code>from sklearn.utils import class_weight
  test = []
  for i in range(500):
          if i&lt;10:
              test = test + [i] * 50
          else:
              test = test + [i] * 1000
  class_weights=class_weight.compute_class_weight(class_weight='balanced',classes=np.unique(test), y=test)
  print(class_weights[0], class_weights[-1])
  </code></pre>
  <p>And you'll get 19.62 ,  0.981. So 10 minority classes have a loss weight of 20, which is enough of a reason for the model to learn to over predict them. Keep in mind though that in principle there's nothing wrong with weighting the loss, in the long run the other classes should be learned as well. If the model really struggle you could try using the focal loss, based on cross entropy but with a different weighting approach.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-25T16:59:33.997Z
Answers.Text: Thanks, @Edoardo. I'll look into focal loss as per your advice. As a follow-up query, does it need a weighing mechanism, or just tuning the ⍺ and  parameters are all that is needed?
Answers.UserDisplayName: helloworld
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-25T11:02:36.093Z
Answers.LastActivityDate: 2022-10-25T11:02:36.093Z
Answers.UserDisplayName: Edoardo Guerriero
Body: <p>I’m working on a classification problem (500 classes). My NN has 3 fully connected layers, followed by an LSTM layer. I use nn.CrossEntropyLoss() as my loss function. To tackle the problem of class imbalance, I use sklearn’s class_weight while initializing the loss</p>
  <pre><code>from sklearn.utils import class_weight
  
  class_weights = class_weight.compute_class_weight(class_weight='balanced', classes=np.unique(y_org), y=y_org)
  
  </code></pre>
  <p>The below plots show the number of predictions made per class</p>
  <p><a href="https://i.sstatic.net/kyC7P.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/kyC7P.png" alt="Distribution of classes (pred vs gt" /></a></p>
  <p>Here, x-axis represents the class and y-axis is the number of times it occurred (as prediction and as ground truth).</p>
  <p>The pattern observed here is that the ground truth and predicted class almost always complement each other. i.e., a class is predicted more times when it’s available less number of times in the ground truth and vice versa (every peak of the blue curve has a dip in orange curve and vice versa). What could be going wrong here?</p>
  
Tags: <deep-learning><classification><pytorch><scikit-learn>
Title: Unexpected behaviour on using class weights in loss
UserDisplayName: helloworld
ThreadText: # Unexpected behaviour on using class weights in loss
  
  (Asked by: username_0 on 2022-10-25)
  
  I’m working on a classification problem (500 classes). My NN has 3 fully connected layers, followed by an LSTM layer. I use nn.CrossEntropyLoss() as my loss function. To tackle the problem of class imbalance, I use sklearn’s class_weight while initializing the loss
  from sklearn.utils import class_weight
  
  class_weights = class_weight.compute_class_weight(class_weight='balanced', classes=np.unique(y_org), y=y_org)
  
  
  The below plots show the number of predictions made per class
  
  Here, x-axis represents the class and y-axis is the number of times it occurred (as prediction and as ground truth).
  The pattern observed here is that the ground truth and predicted class almost always complement each other. i.e., a class is predicted more times when it’s available less number of times in the ground truth and vice versa (every peak of the blue curve has a dip in orange curve and vice versa). What could be going wrong here?
  
  **Tags:** deep-learning, classification, pytorch, scikit-learn
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-10-25. Score: 0 | Accepted answer
  
  In principle nothing looks wrong here. When using weighted cross entropy, the loss is adjusted based on frequency with a weight between 0 and 1. Now normally this works just fine, but since you have 500 classes I suspect your minority classes are boosted a lot. You can actually test it pretty easily.
  from sklearn.utils import class_weight
  test = []
  for i in range(500):
          if i<10:
              test = test + [i] * 50
          else:
              test = test + [i] * 1000
  class_weights=class_weight.compute_class_weight(class_weight='balanced',classes=np.unique(test), y=test)
  print(class_weights[0], class_weights[-1])
  
  And you'll get 19.62 ,  0.981. So 10 minority classes have a loss weight of 20, which is enough of a reason for the model to learn to over predict them. Keep in mind though that in principle there's nothing wrong with weighting the loss, in the long run the other classes should be learned as well. If the model really struggle you could try using the focal loss, based on cross entropy but with a different weighting approach.
  
  > **Comments:**
  > **username_0**: Thanks, @Edoardo. I'll look into focal loss as per your advice. As a follow-up query, does it need a weighing mechanism, or just tuning the ⍺ and  parameters are all that is needed?  
  
  ---
  
  