<p>GANs are notably hard to train and it is not uncommon to have large bumps in the losses. The learning rate is a good start but the instability may come from a wide variety of reasons. I'm assuming that you have no bug in your code or data.</p>

<p>For one, gradient descent is not well suited to the 2-player game we're playing. I've personally found ExtraAdam to yield much more stable training (<a href="https://github.com/GauthierGidel/Variational-Inequality-GAN/blob/master/optim/extragradient.py" rel="nofollow noreferrer">code</a>, <a href="https://arxiv.org/abs/1802.10551" rel="nofollow noreferrer">paper</a>). </p>

<p>It could also come from the loss and many tricks have been developed and one of the most popular one is enforcing smoothness in the gradient (see W-GAN, W-GAN-GP etc.). SpectralNorm (<a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/utils/spectral_norm.py" rel="nofollow noreferrer">code</a> <a href="https://arxiv.org/abs/1802.05957" rel="nofollow noreferrer">paper</a>) is a very popular and recent normalization technique for the discriminator.</p>

<p>There are a number of additional tricks to make GANs work like label smoothing and flipping, different update rates for the discriminator and generators (as in BigGAN for instance). I suggest you have a look at this nice repo of (somewhat seasoned) tricks: <a href="https://github.com/soumith/ganhacks" rel="nofollow noreferrer">ganhacks</a>.</p>

CC BY-SA 4.0
2020-04-08T23:40:30.223Z
2020-04-08T23:40:30.223Z
ted
<p>Sometimes when I am training a DC-GAN on an image dataset, similar to the DC-GAN PyTorch example (<a href="https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html" rel="noreferrer">https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html</a>), either the Generator or Discriminator will get stuck in a large value while the other goes to zero. How should I interpret what is going on right after iteration 1500 in the example loss function image shown below <a href="https://i.sstatic.net/AElrJ.png" rel="noreferrer"><img src="https://i.sstatic.net/AElrJ.png" alt="DC-GAN Training Loss"></a>? Is this an example of mode collapse? Any recommendations for how to make the training more stable? I have tried reducing the learning rate of the Adam optimizer with varying degrees of success. Thanks!</p>

<machine-learning><deep-learning><convolutional-neural-networks><generative-adversarial-networks><generative-model>
What parameters can be tweaked to avoid a generator or discriminator loss collapsing to zero when training a DC-GAN?
jman4162
# What parameters can be tweaked to avoid a generator or discriminator loss collapsing to zero when training a DC-GAN?

(Asked by: username_0 on 2019-06-28)

Sometimes when I am training a DC-GAN on an image dataset, similar to the DC-GAN PyTorch example (<https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html>), either the Generator or Discriminator will get stuck in a large value while the other goes to zero. How should I interpret what is going on right after iteration 1500 in the example loss function image shown below [![DC-GAN Training Loss](https://i.sstatic.net/AElrJ.png)](https://i.sstatic.net/AElrJ.png)? Is this an example of mode collapse? Any recommendations for how to make the training more stable? I have tried reducing the learning rate of the Adam optimizer with varying degrees of success. Thanks!

**Tags:** machine-learning, deep-learning, convolutional-neural-networks, generative-adversarial-networks, generative-model


---

## 1 answers

### Answer by username_1 on 2020-04-08. Score: 2

GANs are notably hard to train and it is not uncommon to have large bumps in the losses. The learning rate is a good start but the instability may come from a wide variety of reasons. I'm assuming that you have no bug in your code or data.

For one, gradient descent is not well suiusername_1 to the 2-player game we're playing. I've personally found ExtraAdam to yield much more stable training ([code](https://github.com/GauthierGidel/Variational-Inequality-GAN/blob/master/optim/extragradient.py), [paper](https://arxiv.org/abs/1802.10551)).

It could also come from the loss and many tricks have been developed and one of the most popular one is enforcing smoothness in the gradient (see W-GAN, W-GAN-GP etc.). SpectralNorm ([code](https://github.com/pytorch/pytorch/blob/master/torch/nn/utils/spectral_norm.py) [paper](https://arxiv.org/abs/1802.05957)) is a very popular and recent normalization technique for the discriminator.

There are a number of additional tricks to make GANs work like label smoothing and flipping, different update rates for the discriminator and generators (as in BigGAN for instance). I suggest you have a look at this nice repo of (somewhat seasoned) tricks: [ganhacks](https://github.com/soumith/ganhacks).


---

