Answers.Body: <p>You already mention quite some good ideas. Unfortunately, what makes GANs difficult to work with is their adversarial training paradigm. There are a ton of very simple and intricate ideas which might improve your training, but they all depend on your data, your exact architecture, your hyperparameters etc. I'll drop some additional links here which should give you some additional ideas on what else you can still do:</p>
  <ol>
  <li><a href="https://www.youtube.com/watch?v=X1mUN6dD8uE" rel="nofollow noreferrer">This video</a> (from the authors of WGAN) gives a detailed overview of some methods you can apply.</li>
  <li><a href="https://github.com/soumith/ganhacks" rel="nofollow noreferrer">This repo</a> provides an overview of methods which you can try.</li>
  </ol>
  <p>Again, some work together, some don't, some might work for others, some might work for you. I recommend just googling for 'tips on training GANs' and seeing what people have experienced. GANs make some high quality stuff and have rapid inference, but sadly they are extremely difficult to train.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-23T12:32:20.387Z
Answers.LastActivityDate: 2023-02-23T12:32:20.387Z
Answers.UserDisplayName: Robin van Hoorn
Body: <p>I am training several different GAN architectures, and I noticed that larger batch sizes may lead to vanishing or exploding gradients. In the interest of accelerating training, however, larger batch sizes are desirable. I learned that weight initialisation schemes such as He/Kaiming initialisation for ReLU-based networks can help alleviate this problem to some degree, as well as adjusting the moments of the Adam optimiser to lower than the default values, such as suggested in the WGAN-GP paper: <a href="https://proceedings.neurips.cc/paper/2017/hash/892c3b1c6dccd52936e27cbd0ff683d6-Abstract.html" rel="nofollow noreferrer">https://proceedings.neurips.cc/paper/2017/hash/892c3b1c6dccd52936e27cbd0ff683d6-Abstract.html</a>. Do you have any other recommendations?</p>
  
Tags: <generative-adversarial-networks><mini-batch-gradient-descent><vanishing-gradient-problem>
Title: How to prevent vanishing/exploding gradients in a GAN with large mini-batch size?
UserDisplayName: postnubilaphoebus
ThreadText: # How to prevent vanishing/exploding gradients in a GAN with large mini-batch size?
  
  (Asked by: username_0 on 2023-02-23)
  
  I am training several different GAN architectures, and I noticed that larger batch sizes may lead to vanishing or exploding gradients. In the interest of accelerating training, however, larger batch sizes are desirable. I learned that weight initialisation schemes such as He/Kaiming initialisation for ReLU-based networks can help alleviate this problem to some degree, as well as adjusting the moments of the Adam optimiser to lower than the default values, such as suggested in the WGAN-GP paper: <https://proceedings.neurips.cc/paper/2017/hash/892c3b1c6dccd52936e27cbd0ff683d6-Abstract.html>. Do you have any other recommendations?
  
  **Tags:** generative-adversarial-networks, mini-batch-gradient-descent, vanishing-gradient-problem
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-02-23. Score: 1 | Accepted answer
  
  You already mention quite some good ideas. Unfortunately, what makes GANs difficult to work with is their adversarial training paradigm. There are a ton of very simple and intricate ideas which might improve your training, but they all depend on your data, your exact architecture, your hyperparameters etc. I'll drop some additional links here which should give you some additional ideas on what else you can still do:
  
  1. [This video](https://www.youtube.com/watch?v=X1mUN6dD8uE) (from the authors of WGAN) gives a detailed overview of some methods you can apply.
  2. [This repo](https://github.com/soumith/ganhacks) provides an overview of methods which you can try.
  
  Again, some work together, some don't, some might work for others, some might work for you. I recommend just googling for 'tips on training GANs' and seeing what people have experienced. GANs make some high quality stuff and have rapid inference, but sadly they are extremely difficult to train.
  
  
  ---
  
  