Answers.Body: <p>@nbro pointed out the paper <a href="https://arxiv.org/pdf/1710.05381.pdf" rel="nofollow noreferrer">A systematic study of the class imbalance problem in convolutional neural networks</a>, which tested class imbalance LeNet for MNIST, on a custom CNN for CIFAR-10, and on ResNet for ImageNet. The paper found that by artificially creating class imbalance on those data sets, the neural networks are significantly deteriorated. The ROC AUC drops by 5-10%, and accuracy decreases by 20-30%. These effects are worsened on more complex tasks.</p>
  <p>There are 3 noteworthy class imbalance approaches to partially alleviate this:</p>
  <ul>
  <li><strong>Undersampling</strong>: sampling the over-represented class less often</li>
  <li><strong>Oversampling</strong>: sampling the under-represented class more often</li>
  <li><strong>Thresholding</strong>: after the neural network learns the weights, during inference, multiply the output class probability by a weight (the class prior, which is different for each class). The weight is the inverse of the class representation of the dataset (i.e. the inverse of <span class="math-container">$\frac{numInstances(c)}{\displaystyle \sum_i numInstances(i)}$</span>, where <span class="math-container">$c$</span> is the current class and <span class="math-container">$numInstances(i)$</span> is the number of unique instances of class <span class="math-container">$i$</span> in the training set.</li>
  </ul>
  <p>The paper concludes with the following best practices:</p>
  <blockquote>
  <p>Regarding the choice of a method to handle CNN training on imbalanced dataset we conclude the following.</p>
  <ul>
  <li><p>The method that in most of the cases outperforms all others with respect to multi-class ROC AUC was oversampling.</p>
  </li>
  <li><p>For extreme ratio of imbalance and large portion of classes being minority, undersampling performs on a par with oversampling. If training time is an issue, undersampling is a better choice in such a scenario since it dramatically reduces the size of the training set</p>
  </li>
  <li><p>To achieve the best accuracy, one should apply thresholding to compensate for prior class probabilities. <strong>A combination of thresholding with baseline and oversampling is the most preferable</strong>, whereas it should not be combined with undersampling.</p>
  </li>
  <li><p>Oversampling should be applied to the level that completely eliminates the imbalance, whereas the optimal undersampling ratio depends on the extent of imbalance. The higher a fraction of minority classes in the imbalanced training set, the more imbalance ratio should be reduced.</p>
  </li>
  <li><p>Oversampling does not cause overfitting of convolutional neural networks, as opposed to some classical machine learning models.</p>
  </li>
  </ul>
  </blockquote>
  <p>The last point is very interesting, because oversampling is known to cause overfitting in classical machine learning models and many have advised against doing it.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-13T02:40:43.747Z
Answers.LastActivityDate: 2020-12-17T00:40:58.557Z
Answers.LastEditDate: 2020-12-17T00:40:58.557Z
Answers.UserDisplayName: user3667125
Body: <p>Before deep learning, I worked with machine learning problems where the data had a large class imbalance (30:1 or worse ratios). At that time, all the classifiers struggled, even after under-sampling the represented classes and creating synthetic examples of the underrepresented classes -- except Random Forest, which was a bit more robust than the others, but still not great.</p>
  <p>What are guidelines for class distribution when it comes to deep learning (CNNs, ResNets, transformers, etc)? Must the representation of each class be 1:1? Or maybe it's &quot;good enough&quot; as long as it is under some ratio like 2:1? Or is deep learning completely immune to class imbalance as long as we have enough training data?</p>
  <p>Furthermore, as a general guideline, should each class have a certain minimum number of training examples (maybe some multiple of the number of weights of the network)?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-12T12:22:56.793Z
Comments.Text: I don't think that neural networks are immune to class imbalance, but I'm not really an expert on this topic. One way to deal with class imbalance could be to make the NN [uncertainty-aware](https://openaccess.thecvf.com/content_CVPR_2019/papers/Khan_Striking_the_Right_Balance_With_Uncertainty_CVPR_2019_paper.pdf). Maybe you could look at [this paper](https://www.sciencedirect.com/science/article/pii/S0893608018302107). Feel free to provide an answer to your own question, once you know the answer, if nobody provides an answer meanwhile.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-13T00:49:58.577Z
Comments.Text: Thanks! Will give it a read and post an answer later
Comments.UserDisplayName: user3667125
Tags: <neural-networks><deep-learning><datasets><data-preprocessing><imbalanced-datasets>
Title: How robust are deep networks to class imbalance?
UserDisplayName: user3667125
ThreadText: # How robust are deep networks to class imbalance?
  
  (Asked by: username_0 on 2020-12-12)
  
  Before deep learning, I worked with machine learning problems where the data had a large class imbalance (30:1 or worse ratios). At that time, all the classifiers struggled, even after under-sampling the represented classes and creating synthetic examples of the underrepresented classes -- except Random Forest, which was a bit more robust than the others, but still not great.
  
  What are guidelines for class distribution when it comes to deep learning (CNNs, ResNets, transformers, etc)? Must the representation of each class be 1:1? Or maybe it's "good enough" as long as it is under some ratio like 2:1? Or is deep learning completely immune to class imbalance as long as we have enough training data?
  
  Furthermore, as a general guideline, should each class have a certain minimum number of training examples (maybe some multiple of the number of weights of the network)?
  
  **Tags:** neural-networks, deep-learning, datasets, data-preprocessing, imbalanced-datasets
  > **Comments:**
  > **username_1**: I don't think that neural networks are immune to class imbalance, but I'm not really an expert on this topic. One way to deal with class imbalance could be to make the NN [uncertainty-aware](https://openaccess.thecvf.com/content_CVPR_2019/papers/Khan_Striking_the_Right_Balance_With_Uncertainty_CVPR_2019_paper.pdf). Maybe you could look at [this paper](https://www.sciencedirect.com/science/article/pii/S0893608018302107). Feel free to provide an answer to your own question, once you know the answer, if nobody provides an answer meanwhile.  
  > **username_0**: Thanks! Will give it a read and post an answer later  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2020-12-13. Score: 2 | Accepted answer
  
  @nbro pointed out the paper [A systematic study of the class imbalance problem in convolutional neural networks](https://arxiv.org/pdf/1710.05381.pdf), which tested class imbalance LeNet for MNIST, on a custom CNN for CIFAR-10, and on ResNet for ImageNet. The paper found that by artificially creating class imbalance on those data sets, the neural networks are significantly deteriorated. The ROC AUC drops by 5-10%, and accuracy decreases by 20-30%. These effects are worsened on more complex tasks.
  
  There are 3 noteworthy class imbalance approaches to partially alleviate this:
  
  * **Undersampling**: sampling the over-represented class less often
  * **Oversampling**: sampling the under-represented class more often
  * **Thresholding**: after the neural network learns the weights, during inference, multiply the output class probability by a weight (the class prior, which is different for each class). The weight is the inverse of the class representation of the dataset (i.e. the inverse of $\frac{numInstances(c)}{\displaystyle \sum\_i numInstances(i)}$, where $c$ is the current class and $numInstances(i)$ is the number of unique instances of class $i$ in the training set.
  
  The paper concludes with the following best practices:
  
  > Regarding the choice of a method to handle CNN training on imbalanced dataset we conclude the following.
  >
  > * The method that in most of the cases outperforms all others with respect to multi-class ROC AUC was oversampling.
  > * For extreme ratio of imbalance and large portion of classes being minority, undersampling performs on a par with oversampling. If training time is an issue, undersampling is a better choice in such a scenario since it dramatically reduces the size of the training set
  > * To achieve the best accuracy, one should apply thresholding to compensate for prior class probabilities. **A combination of thresholding with baseline and oversampling is the most preferable**, whereas it should not be combined with undersampling.
  > * Oversampling should be applied to the level that completely eliminates the imbalance, whereas the optimal undersampling ratio depends on the extent of imbalance. The higher a fraction of minority classes in the imbalanced training set, the more imbalance ratio should be reduced.
  > * Oversampling does not cause overfitting of convolutional neural networks, as opposed to some classical machine learning models.
  
  The last point is very interesting, because oversampling is known to cause overfitting in classical machine learning models and many have advised against doing it.
  
  
  ---
  
  