Answers.Body: <p>If you start with perpect discriminator, loss function will be saturated, and gradient of loss will be very small, so feedback for the generator also will be small, and learning will be slow down as a result. Actually, it is allways desired for discriminator and generator to learn balancedly. Additionally, it is claimed that Wasserstein Loss take care of this problem.</p>
  <p>You can find more information in <a href="https://lilianweng.github.io/lil-log/2017/08/20/from-GAN-to-WGAN.html#vanishing-gradient" rel="noreferrer">this article</a>. (I strongly suggest to read)</p>
  <p>Also from the paper &quot; <a href="https://arxiv.org/abs/1701.04862" rel="noreferrer">Towards principled methods for training generative adversarial networks</a>&quot;:</p>
  <blockquote>
  <p>In theory, one would expect therefore that we would first train the discriminator as close as we can to optimality (so the
  cost function on <span class="math-container">$\theta$</span> better approximates the JSD), and then do gradient steps on  <span class="math-container">$\theta$</span>, alternating these
  two things. However, this doesn’t work. In practice, as the discriminator gets better, the updates
  to the generator get consistently worse. The original GAN paper argued that this issue arose from
  saturation, and switched to another similar cost function that doesn’t have this problem. However,
  even with this new cost function, updates tend to get worse and optimization gets massively unstable.</p>
  </blockquote>
  <p>Note 1: <span class="math-container">$\theta$</span> is parameters of generator</p>
  <p>Note 2: JSD: Jensen–Shannon divergence. For optimal discrimantor, loss is equal to <span class="math-container">$L(D^{ \ast }, g_{\theta})=  2 JSD(P_{r}, P_{g}) - 2log(2)$</span></p>
  <p>Also from the paper of <a href="https://arxiv.org/abs/1701.07875" rel="noreferrer">Wasserstein Loss</a>:</p>
  <blockquote>
  <p>The fact that the EM distance is continuous and differentiable a.e. means that
  we can (and should) train the critic till optimality. The argument is simple, the more we train the critic, the more reliable gradient of the Wasserstein we get, which is actually useful by the fact that Wasserstein is differentiable almost everywhere. For the JS, as the discriminator gets better the gradients get more reliable but the true gradient is 0 since the JS is locally saturated and we get vanishing gradients ... In Figure 2 we show a proof of concept of this, where we train a GAN discriminator and a WGAN critic till optimality. The discriminator learns very quickly to distinguish between fake and real, and as expected provides no reliable gradient information. The critic, however, can’t saturate, and converges to a linear function that gives remarkably clean gradients everywhere.
  <a href="https://i.sstatic.net/sZPDK.jpg" rel="noreferrer"><img src="https://i.sstatic.net/sZPDK.jpg" alt="Figure 2" /></a></p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-24T20:45:49.543Z
Answers.Text: I find this a very good answer. However, I wonder if relativistic GANs wouldn't cope with this and even benefit from the pretrained discriminator.
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-24T22:08:33.893Z
Answers.Text: @David Thanks for the comment. I honestly heared Relativistic GAN first time from you. Seems interesting paper.
Answers.UserDisplayName: Enes
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-20T23:09:27.573Z
Answers.LastActivityDate: 2020-12-24T16:47:38.163Z
Answers.LastEditDate: 2020-12-24T16:47:38.163Z
Answers.UserDisplayName: Enes
Body: <p>In many implementations/tutorials of GANs that I've seen so far (e.g. <a href="https://github.com/eriklindernoren/Keras-GAN/blob/master/dcgan/dcgan.py" rel="noreferrer">this</a>), the generator and discriminator start with no prior knowledge. They continuously improve their performance with training. This makes me wonder — is it possible to use a pre-trained discriminator? I have two motivations for doing so:</p>
  <ol>
  <li>Eliminating the overhead of training the discriminator</li>
  <li>Being able to use already existing cool models</li>
  </ol>
  <p>Would the generator be able to learn just the same, or is it dependent on the fact that they start from scratch?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-17T21:05:55.833Z
Comments.Text: Some pretty interesting comments on the subject here: https://www.reddit.com/r/MachineLearning/comments/dwzb81/r_training_gans_with_a_pretrained_discriminator/
Comments.UserDisplayName: tester
Tags: <convolutional-neural-networks><generative-adversarial-networks>
Title: Can I start with perfect discriminator in GAN?
UserDisplayName: mark mark
ThreadText: # Can I start with perfect discriminator in GAN?
  
  (Asked by: username_0 on 2020-12-15)
  
  In many implementations/tutorials of GANs that I've seen so far (e.g. [this](https://github.com/eriklindernoren/Keras-GAN/blob/master/dcgan/dcgan.py)), the generator and discriminator start with no prior knowledge. They continuously improve their performance with training. This makes me wonder — is it possible to use a pre-trained discriminator? I have two motivations for doing so:
  
  1. Eliminating the overhead of training the discriminator
  2. Being able to use already existing cool models
  
  Would the generator be able to learn just the same, or is it dependent on the fact that they start from scratch?
  
  **Tags:** convolutional-neural-networks, generative-adversarial-networks
  
  > **Comments:**
  > **username_1**: Some pretty interesting comments on the subject here: https://www.reddit.com/r/MachineLearning/comments/dwzb81/r_training_gans_with_a_pretrained_discriminator/  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-12-20. Score: 7
  
  If you start with perpect discriminator, loss function will be saturated, and gradient of loss will be very small, so feedback for the generator also will be small, and learning will be slow down as a result. Actually, it is allways desired for discriminator and generator to learn balancedly. Additionally, it is claimed that Wasserstein Loss take care of this problem.
  
  You can find more information in [this article](https://lilianweng.github.io/lil-log/2017/08/20/from-GAN-to-WGAN.html#vanishing-gradient). (I strongly suggest to read)
  
  Also from the paper " [Towards principled methods for training generative adversarial networks](https://arxiv.org/abs/1701.04862)":
  
  > In theory, one would expect therefore that we would first train the discriminator as close as we can to optimality (so the
  > cost function on $\theta$ better approximates the JSD), and then do gradient steps on $\theta$, alternating these
  > two things. However, this doesn’t work. In practice, as the discriminator gets better, the updates
  > to the generator get consistently worse. The original GAN paper argued that this issue arose from
  > saturation, and switched to another similar cost function that doesn’t have this problem. However,
  > even with this new cost function, updates tend to get worse and optimization gets massively unstable.
  
  Note 1: $\theta$ is parameters of generator
  
  Note 2: JSD: Jensen–Shannon divergence. For optimal discrimantor, loss is equal to $L(D^{ \ast }, g\_{\theta})= 2 JSD(P\_{r}, P\_{g}) - 2log(2)$
  
  Also from the paper of [Wasserstein Loss](https://arxiv.org/abs/1701.07875):
  
  > The fact that the EM distance is continuous and differentiable a.e. means that
  > we can (and should) train the critic till optimality. The argument is simple, the more we train the critic, the more reliable gradient of the Wasserstein we get, which is actually useful by the fact that Wasserstein is differentiable almost everywhere. For the JS, as the discriminator gets better the gradients get more reliable but the true gradient is 0 since the JS is locally saturated and we get vanishing gradients ... In Figure 2 we show a proof of concept of this, where we train a GAN discriminator and a WGAN critic till optimality. The discriminator learns very quickly to distinguish between fake and real, and as expected provides no reliable gradient information. The critic, however, can’t saturate, and converges to a linear function that gives remarkably clean gradients everywhere.
  > [![Figure 2](https://i.sstatic.net/sZPDK.jpg)](https://i.sstatic.net/sZPDK.jpg)
  
  > **Comments:**
  > **username_3**: I find this a very good answer. However, I wonder if relativistic GANs wouldn't cope with this and even benefit from the pretrained discriminator.  
  > **username_2**: @username_3 Thanks for the comment. I honestly heared Relativistic GAN first time from you. Seems interesting paper.  
  
  ---
  
  