Answers.Body: <p>GANs are notably hard to train and it is not uncommon to have large bumps in the losses. The learning rate is a good start but the instability may come from a wide variety of reasons. I'm assuming that you have no bug in your code or data.</p>
  
  <p>For one, gradient descent is not well suited to the 2-player game we're playing. I've personally found ExtraAdam to yield much more stable training (<a href="https://github.com/GauthierGidel/Variational-Inequality-GAN/blob/master/optim/extragradient.py" rel="nofollow noreferrer">code</a>, <a href="https://arxiv.org/abs/1802.10551" rel="nofollow noreferrer">paper</a>). </p>
  
  <p>It could also come from the loss and many tricks have been developed and one of the most popular one is enforcing smoothness in the gradient (see W-GAN, W-GAN-GP etc.). SpectralNorm (<a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/utils/spectral_norm.py" rel="nofollow noreferrer">code</a> <a href="https://arxiv.org/abs/1802.05957" rel="nofollow noreferrer">paper</a>) is a very popular and recent normalization technique for the discriminator.</p>
  
  <p>There are a number of additional tricks to make GANs work like label smoothing and flipping, different update rates for the discriminator and generators (as in BigGAN for instance). I suggest you have a look at this nice repo of (somewhat seasoned) tricks: <a href="https://github.com/soumith/ganhacks" rel="nofollow noreferrer">ganhacks</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-08T23:40:30.223Z
Answers.LastActivityDate: 2020-04-08T23:40:30.223Z
Answers.UserDisplayName: ted
Body: <p>Sometimes when I am training a DC-GAN on an image dataset, similar to the DC-GAN PyTorch example (<a href="https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html" rel="noreferrer">https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html</a>), either the Generator or Discriminator will get stuck in a large value while the other goes to zero. How should I interpret what is going on right after iteration 1500 in the example loss function image shown below <a href="https://i.sstatic.net/AElrJ.png" rel="noreferrer"><img src="https://i.sstatic.net/AElrJ.png" alt="DC-GAN Training Loss"></a>? Is this an example of mode collapse? Any recommendations for how to make the training more stable? I have tried reducing the learning rate of the Adam optimizer with varying degrees of success. Thanks!</p>
  
Tags: <machine-learning><deep-learning><convolutional-neural-networks><generative-adversarial-networks><generative-model>
Title: What parameters can be tweaked to avoid a generator or discriminator loss collapsing to zero when training a DC-GAN?
UserDisplayName: jman4162
ThreadText: # What parameters can be tweaked to avoid a generator or discriminator loss collapsing to zero when training a DC-GAN?
  
  (Asked by: username_0 on 2019-06-28)
  
  Sometimes when I am training a DC-GAN on an image dataset, similar to the DC-GAN PyTorch example (<https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html>), either the Generator or Discriminator will get stuck in a large value while the other goes to zero. How should I interpret what is going on right after iteration 1500 in the example loss function image shown below [![DC-GAN Training Loss](https://i.sstatic.net/AElrJ.png)](https://i.sstatic.net/AElrJ.png)? Is this an example of mode collapse? Any recommendations for how to make the training more stable? I have tried reducing the learning rate of the Adam optimizer with varying degrees of success. Thanks!
  
  **Tags:** machine-learning, deep-learning, convolutional-neural-networks, generative-adversarial-networks, generative-model
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-04-08. Score: 2
  
  GANs are notably hard to train and it is not uncommon to have large bumps in the losses. The learning rate is a good start but the instability may come from a wide variety of reasons. I'm assuming that you have no bug in your code or data.
  
  For one, gradient descent is not well suited to the 2-player game we're playing. I've personally found ExtraAdam to yield much more stable training ([code](https://github.com/GauthierGidel/Variational-Inequality-GAN/blob/master/optim/extragradient.py), [paper](https://arxiv.org/abs/1802.10551)).
  
  It could also come from the loss and many tricks have been developed and one of the most popular one is enforcing smoothness in the gradient (see W-GAN, W-GAN-GP etc.). SpectralNorm ([code](https://github.com/pytorch/pytorch/blob/master/torch/nn/utils/spectral_norm.py) [paper](https://arxiv.org/abs/1802.05957)) is a very popular and recent normalization technique for the discriminator.
  
  There are a number of additional tricks to make GANs work like label smoothing and flipping, different update rates for the discriminator and generators (as in BigGAN for instance). I suggest you have a look at this nice repo of (somewhat seasoned) tricks: [ganhacks](https://github.com/soumith/ganhacks).
  
  
  ---
  
  