Answers.Body: <p>This comes from the fact that you want the same magnitude from the loss. Think of it this way: a non-weighted loss function actually has all its weights to 1 and so over the whole data set, samples are weighted with 1 and the sum of all weights is therefore <span class="math-container">$N$</span>, if <span class="math-container">$N$</span> is the total number of samples.</p>
  
  <p>Now in the case of a weighted loss, we want the weights to also sum to <span class="math-container">$N$</span> so that the loss's magnitude is comparable (<span class="math-container">$i = 1..C$</span> are your classes, <span class="math-container">$N_i$</span> is the number of samples for class <span class="math-container">$i$</span>):</p>
  
  <p><span class="math-container">$$S = \sum_{i=0\ }^{C} \sum_{s_i=1}^{N_i} w_{i} = \sum_{i=0}^{i}\sum_{s_i=1}^{N_i}\frac{1}{N_i} \frac{N}{C} = \frac{N}{C} \sum_{i=0}^{C}\sum_{s_i=1}^{N_i}\frac{1}{N_i} = \frac{N}{C} \sum_{i=0}^{C}N_i\frac{1}{N_i}  = \frac{N}{C} \sum_{i=0}^{C}1 = \frac{N}{C} C = N$$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-14T13:10:03.737Z
Answers.LastActivityDate: 2020-04-14T13:10:03.737Z
Answers.UserDisplayName: ted
Body: <p>I am trying to build a classifier which should be trained with the cross entropy loss. The training data is highly class-imbalanced. To tackle this, I've gone through the <a href="https://www.tensorflow.org/tutorials/structured_data/imbalanced_data#calculate_class_weights" rel="nofollow noreferrer">advice of the tensorflow docs</a></p>
  
  <p>and now I am using a weighted cross entropy loss where the weights are calculated as </p>
  
  <pre><code>weight_for_class_a = (1 / samples_for_class_a) * total_number_of_samples/number_of_classes
  </code></pre>
  
  <p>following the mentioned tutorial. </p>
  
  <p>It works perfectly, but why is there this factor <code>total_number_of_samples/number_of_classes</code>?
  The mentioned tutorial says this</p>
  
  <blockquote>
    <p>[...] helps keep the loss to a similar magnitude.</p>
  </blockquote>
  
  <p>But I don not understand why. Can someone clarify?</p>
  
Tags: <machine-learning><classification><datasets><weights><cross-entropy>
Title: How are weights for weighted x-entropy loss on imbalanced data calculated?
UserDisplayName: jmatin
ThreadText: # How are weights for weighted x-entropy loss on imbalanced data calculated?
  
  (Asked by: username_0 on 2020-04-14)
  
  I am trying to build a classifier which should be trained with the cross entropy loss. The training data is highly class-imbalanced. To tackle this, I've gone through the [advice of the tensorflow docs](https://www.tensorflow.org/tutorials/structured_data/imbalanced_data#calculate_class_weights)
  
  and now I am using a weighted cross entropy loss where the weights are calculated as
  
  ```
  weight_for_class_a = (1 / samples_for_class_a) * total_number_of_samples/number_of_classes
  
  ```
  
  following the mentioned tutorial.
  
  It works perfectly, but why is there this factor `total_number_of_samples/number_of_classes`?
  The mentioned tutorial says this
  
  > [...] helps keep the loss to a similar magnitude.
  
  But I don not understand why. Can someone clarify?
  
  **Tags:** machine-learning, classification, datasets, weights, cross-entropy
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-04-14. Score: 0
  
  This comes from the fact that you want the same magnitude from the loss. Think of it this way: a non-weighted loss function actually has all its weights to 1 and so over the whole data set, samples are weighted with 1 and the sum of all weights is therefore $N$, if $N$ is the total number of samples.
  
  Now in the case of a weighted loss, we want the weights to also sum to $N$ so that the loss's magnitude is comparable ($i = 1..C$ are your classes, $N\_i$ is the number of samples for class $i$):
  
  $$S = \sum\_{i=0\ }^{C} \sum\_{s\_i=1}^{N\_i} w\_{i} = \sum\_{i=0}^{i}\sum\_{s\_i=1}^{N\_i}\frac{1}{N\_i} \frac{N}{C} = \frac{N}{C} \sum\_{i=0}^{C}\sum\_{s\_i=1}^{N\_i}\frac{1}{N\_i} = \frac{N}{C} \sum\_{i=0}^{C}N\_i\frac{1}{N\_i} = \frac{N}{C} \sum\_{i=0}^{C}1 = \frac{N}{C} C = N$$
  
  
  ---
  
  