Answers.Body: <p>Balancing the training of the generator and the critic is essential for high-quality GANs. If either the generator or the discriminator overpowers the other, the model will not converge to a satisfactory state.</p>
  <p>Balancing the generator and the discriminator can be done in many different ways. For example, you could make the generator bigger, giving it more computational capacity, but potentially slowing down its convergence because there are more parameters to tune.</p>
  <p>The way you state, to update either the generator or the discriminator more than the other, is another way of trying to achieve such a balance. Although some tutorials of normal GANs might not show that this balancing is necessary (as they try to make the tutorial as easy as possible), balancing a traditional GAN by training one module more than the other is very common.</p>
  <p>With WGAN-GP, we penalize the discriminator which hence learns less fast. As a result, the assumption is that we need to train it more to stay in balance. Consequently, the paper proposes the balancing hyperparameter that it does.</p>
  <p>Concluding, this balancing parameter can also be added to normal GANs, or any GAN for that matter. The paper proposes a hyperparameter setting indicative for their test results. However, please play around with it if you want to apply it for your specific application. I've trained GANs both with training the generator 10x more than the discriminator and also the other way around. There is no golden rule for this kind of stuff.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-07T12:06:20.117Z
Answers.Text: My [tutorial/implementation of the traditional GAN](https://github.com/rdvhoorn/GAN-Pytorch) does feature this hyperparameter. Its also a bit more complete than the tensorflow implementation.
Answers.UserDisplayName: Robin van Hoorn
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-09T13:18:04.100Z
Answers.Text: Thanks for the explanation! Although it is often stated that GANs are difficult to balance, indeed many tutorials do not elaborate on this hyperparameter.
Answers.UserDisplayName: Value_Investor
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-07T12:04:48.623Z
Answers.LastActivityDate: 2023-07-07T12:04:48.623Z
Answers.UserDisplayName: Robin van Hoorn
Body: <p>As noted in <a href="https://arxiv.org/abs/1704.00028" rel="nofollow noreferrer">the paper introducing WGAN-GP</a> (see the pseudo-code), for each minibatch of data, the generator's weights are updated only once, and the critic (or discriminator) is updated multiple times. This is also evident from <a href="https://keras.io/examples/generative/wgan_gp/#create-the-wgangp-model:%7E:text=for%20i%20in%20range(self.d_steps)%3A" rel="nofollow noreferrer">the Keras implementation</a>. However, normal GANs update both the generator and the discriminator only once per batch, as is demonstrated in <a href="https://www.tensorflow.org/tutorials/generative/dcgan#define_the_training_loop" rel="nofollow noreferrer">the TensorFlow implementation</a>. Why this difference?</p>
  
Tags: <deep-learning><generative-adversarial-networks>
Title: Why does the critic of WGAN-GP run more steps than the generator?
UserDisplayName: Value_Investor
ThreadText: # Why does the critic of WGAN-GP run more steps than the generator?
  
  (Asked by: username_0 on 2023-07-07)
  
  As noted in [the paper introducing WGAN-GP](https://arxiv.org/abs/1704.00028) (see the pseudo-code), for each minibatch of data, the generator's weights are updated only once, and the critic (or discriminator) is updated multiple times. This is also evident from [the Keras implementation](https://keras.io/examples/generative/wgan_gp/#create-the-wgangp-model:%7E:text=for%20i%20in%20range(self.d_steps)%3A). However, normal GANs update both the generator and the discriminator only once per batch, as is demonstrated in [the TensorFlow implementation](https://www.tensorflow.org/tutorials/generative/dcgan#define_the_training_loop). Why this difference?
  
  **Tags:** deep-learning, generative-adversarial-networks
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-07-07. Score: 1 | Accepted answer
  
  Balancing the training of the generator and the critic is essential for high-quality GANs. If either the generator or the discriminator overpowers the other, the model will not converge to a satisfactory state.
  
  Balancing the generator and the discriminator can be done in many different ways. For example, you could make the generator bigger, giving it more computational capacity, but potentially slowing down its convergence because there are more parameters to tune.
  
  The way you state, to update either the generator or the discriminator more than the other, is another way of trying to achieve such a balance. Although some tutorials of normal GANs might not show that this balancing is necessary (as they try to make the tutorial as easy as possible), balancing a traditional GAN by training one module more than the other is very common.
  
  With WGAN-GP, we penalize the discriminator which hence learns less fast. As a result, the assumption is that we need to train it more to stay in balance. Consequently, the paper proposes the balancing hyperparameter that it does.
  
  Concluding, this balancing parameter can also be added to normal GANs, or any GAN for that matter. The paper proposes a hyperparameter setting indicative for their test results. However, please play around with it if you want to apply it for your specific application. I've trained GANs both with training the generator 10x more than the discriminator and also the other way around. There is no golden rule for this kind of stuff.
  
  > **Comments:**
  > **username_1**: My [tutorial/implementation of the traditional GAN](https://github.com/rdvhoorn/GAN-Pytorch) does feature this hyperparameter. Its also a bit more complete than the tensorflow implementation.  
  > **username_0**: Thanks for the explanation! Although it is often stated that GANs are difficult to balance, indeed many tutorials do not elaborate on this hyperparameter.  
  
  ---
  
  