Answers.Body: <p>In my experience, GANs work really well for the scenario of semi-supervised learning, where you don't necessarily have labels for all your class <span class="math-container">$B$</span> data, but you do have a balanced dataset. In my (limited) experience, you do have to have a balanced (in numbers) set of <span class="math-container">$A$</span> and <span class="math-container">$B$</span> objects, even though you are not sure of the labels.</p>
  <p>And yes, GANs can overfit to outliers as well, especially in the absence of a lot of examples, so be cautious.</p>
  <p>Currently, the version that works best for me (in terms of GANs) is WGAN-GP or WGAN-LP in combination with Optimistic Mirror Descent Adam (<a href="https://discuss.pytorch.org/t/optimistic-mirror-descent-in-saddle-point-problems-adam-optimizer-modification/26928" rel="nofollow noreferrer">here</a>, Ncritic/Nactor = 1). Take a look at the paper by <a href="http://cs231n.stanford.edu/reports/2017/pdfs/313.pdf" rel="nofollow noreferrer">Adiwardana et al.</a>, especially Fig. 7 (astonishing!), for semi-supervised learning with a limited number of class labels.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-11-02T02:30:27.607Z
Answers.LastActivityDate: 2021-01-17T20:21:52.940Z
Answers.LastEditDate: 2021-01-17T20:21:52.940Z
Answers.UserDisplayName: Foivos
Body: <h3>Problem setting</h3>
  <p>We have to do a binary classification of data given a training dataset <span class="math-container">$D$</span>, where most items belong to class <span class="math-container">$A$</span> and some items belong to class <span class="math-container">$B$</span>, so the classes are heavily imbalanced.</p>
  <h3>Approach</h3>
  <p>We wanted to use a GAN to produce more samples of class <span class="math-container">$B$</span>, so that our final classification model has a nearly balanced set to train.</p>
  <h3>Problem</h3>
  <p>Let's say that the data from both classes <span class="math-container">$A$</span> and <span class="math-container">$B$</span> are very similar. Given that we want to produce synthetic data with class <span class="math-container">$B$</span> with the GAN, we feed real <span class="math-container">$B$</span> samples that we have into the discriminator alongside with generated samples. However, <span class="math-container">$A$</span> and <span class="math-container">$B$</span> are similar. It could happen that the generator produces an item <span class="math-container">$x$</span>, that would naturally belong to class <span class="math-container">$A$</span>. But since the discriminator has never seen class-<span class="math-container">$A$</span> items before and both classes are very close, the discriminator could say that this item <span class="math-container">$x$</span> is part of the original data that was fed into the discriminator. So, the generator successfully fooled the discriminator in believing that an item <span class="math-container">$x$</span> is part of the original data of class <span class="math-container">$B$</span>, while <span class="math-container">$x$</span> is actually part of class <span class="math-container">$A$</span>.</p>
  <p>If the GAN keeps producing items like this, the produced data is useless, since it would add heavy noise to the original data, if combined.</p>
  <p>At the same time, let's say before we start training the generator, we show the discriminator our classes <span class="math-container">$A$</span> and <span class="math-container">$B$</span> samples while giving information, that the class-<span class="math-container">$A$</span> items are not part of class <span class="math-container">$B$</span> (through backprop). The discriminator would learn to reject class-<span class="math-container">$A$</span> items that are fed to it. But wouldn't this mean that the discriminator has just become the classification model we wanted to build in the first place to distinguish between class <span class="math-container">$A$</span> and class <span class="math-container">$B$</span>?</p>
  <p>Do you know any solution to the above-stated problem or can you refer to some paper/other posts on this?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-10-27T16:57:16.863Z
Comments.Text: You cannot really use GANs like this to "bootstrap" or augment a dataset for supervised learning. The quality of your generated samples is just as limited by the lack of data as just training direct on the supervised learning set - really it is no different in concept to using the labels from one supervised learner to train another. Or mounting a fan on a sailing boat to blow into the sail . . .
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-10-28T01:17:29.933Z
Comments.Text: @NeilSlater Do you think it is possible to use Conditional GANs as proposed by Mirza and Osindero?
Comments.UserDisplayName: frederik
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-10-28T09:26:04.203Z
Comments.Text: I doubt it is possible, but I have not read the work on Conditional GANs. I suspect that about the only thing you would achieve is some regularisation (which may help prevent over-fitting), but other regularisation techniques would be easier to implement than using generative models in your case IMO. If you can find some way to inject and use far more data into your generative model than you have available for the supervised model, then *that* might change things for you, and make it worthwhile.
Comments.UserDisplayName: Neil Slater
Tags: <neural-networks><generative-adversarial-networks><binary-classification><imbalanced-datasets>
Title: How can I use Generative Adversarial Networks to solve the imbalanced class problem?
UserDisplayName: frederik
ThreadText: # How can I use Generative Adversarial Networks to solve the imbalanced class problem?
  
  (Asked by: username_0 on 2018-10-27)
  
  ### Problem setting
  
  We have to do a binary classification of data given a training dataset $D$, where most items belong to class $A$ and some items belong to class $B$, so the classes are heavily imbalanced.
  
  ### Approach
  
  We wanted to use a GAN to produce more samples of class $B$, so that our final classification model has a nearly balanced set to train.
  
  ### Problem
  
  Let's say that the data from both classes $A$ and $B$ are very similar. Given that we want to produce synthetic data with class $B$ with the GAN, we feed real $B$ samples that we have into the discriminator alongside with generated samples. However, $A$ and $B$ are similar. It could happen that the generator produces an item $x$, that would naturally belong to class $A$. But since the discriminator has never seen class-$A$ items before and both classes are very close, the discriminator could say that this item $x$ is part of the original data that was fed into the discriminator. So, the generator successfully fooled the discriminator in believing that an item $x$ is part of the original data of class $B$, while $x$ is actually part of class $A$.
  
  If the GAN keeps producing items like this, the produced data is useless, since it would add heavy noise to the original data, if combined.
  
  At the same time, let's say before we start training the generator, we show the discriminator our classes $A$ and $B$ samples while giving information, that the class-$A$ items are not part of class $B$ (through backprop). The discriminator would learn to reject class-$A$ items that are fed to it. But wouldn't this mean that the discriminator has just become the classification model we wanted to build in the first place to distinguish between class $A$ and class $B$?
  
  Do you know any solution to the above-stated problem or can you refer to some paper/other posts on this?
  
  **Tags:** neural-networks, generative-adversarial-networks, binary-classification, imbalanced-datasets
  
  > **Comments:**
  > **username_1**: You cannot really use GANs like this to "bootstrap" or augment a dataset for supervised learning. The quality of your generated samples is just as limited by the lack of data as just training direct on the supervised learning set - really it is no different in concept to using the labels from one supervised learner to train another. Or mounting a fan on a sailing boat to blow into the sail . . .  
  > **username_0**: @username_1 Do you think it is possible to use Conditional GANs as proposed by Mirza and Osindero?  
  > **username_1**: I doubt it is possible, but I have not read the work on Conditional GANs. I suspect that about the only thing you would achieve is some regularisation (which may help prevent over-fitting), but other regularisation techniques would be easier to implement than using generative models in your case IMO. If you can find some way to inject and use far more data into your generative model than you have available for the supervised model, then *that* might change things for you, and make it worthwhile.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2018-11-02. Score: 1 | Accepted answer
  
  In my experience, GANs work really well for the scenario of semi-supervised learning, where you don't necessarily have labels for all your class $B$ data, but you do have a balanced dataset. In my (limited) experience, you do have to have a balanced (in numbers) set of $A$ and $B$ objects, even though you are not sure of the labels.
  
  And yes, GANs can overfit to outliers as well, especially in the absence of a lot of examples, so be cautious.
  
  Currently, the version that works best for me (in terms of GANs) is WGAN-GP or WGAN-LP in combination with Optimistic Mirror Descent Adam ([here](https://discuss.pytorch.org/t/optimistic-mirror-descent-in-saddle-point-problems-adam-optimizer-modification/26928), Ncritic/Nactor = 1). Take a look at the paper by [Adiwardana et al.](http://cs231n.stanford.edu/reports/2017/pdfs/313.pdf), especially Fig. 7 (astonishing!), for semi-supervised learning with a limited number of class labels.
  
  
  ---
  
  