Body: <p>I'm currently trying to train a GAN to recreate similar images from a dataset. The dataset is using the Eiffel Tower Pictures from Googles <a href="https://github.com/googlecreativelab/quickdraw-dataset" rel="nofollow noreferrer">Quick Draw dataset</a>. The images aren't very large (only 12x12 pixels) and are all black and white.</p>
  <p>The performance increases at an expected rate initially however after a certain point the quality begins to go down despite the cost from both generator and discriminator networks seeming to stay at a steady value as expected.</p>
  <p><a href="https://i.sstatic.net/rK1PL.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/rK1PL.png" alt="Good Quality Image" /></a>
  <a href="https://i.sstatic.net/ZD8BV.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/ZD8BV.png" alt="Bad Quality Image" /></a>
  <a href="https://i.sstatic.net/LzYmI.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/LzYmI.png" alt="Worst Quality Image" /></a></p>
  <p>I've tried changing the learning rate and other hyper parameters however they all end up with the same result of the network eventually getting worse.</p>
  <p>My learning rate is currently 0.01 which I'm aware is quite high compared to what other people are using but anything lower takes too long to train and the results don't seem any better even if I do give it long enough.</p>
  <p>Any pointers on what could be causing this or the specific name of this problem if it's a common issue with GANs would be appreciated.</p>
  <p>Thanks</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-12-31T13:30:37.257Z
Comments.Text: Which adversarial loss are you using? Also, is the drop in quality constant in time or do the images get worse/better/worse/better? It might be that the high learning rate push the weights out of the global minima after reaching it, in which case a scheduler or dynamic decrease of learning rate would be enough to solve the problem.
Comments.UserDisplayName: Edoardo Guerriero
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-01T18:11:39.660Z
Comments.Text: Not quite sure what you mean by adversarial loss, but if you mean the loss function used by the generator and discriminator, it's the mean square error/quadratic loss function. The drop in quality seems to follow the pattern that pixels in the images produced later on get gradually blacker until all pixels are either completely black or white instead of also grey which produces worse images and doesn't get better. I'll try using a scheduler for the learning rate although the window of cycles in which good quality images are produced is quite slim so I'm not so sure of this working.
Comments.UserDisplayName: user17361867
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-05-28T07:25:59.533Z
Comments.Text: were you ever able to get it working? I am having the same issue where the generated images get better after a while, but just keep getting worse and worse. I monitored the gradient coming into the generator for the picture of "3":[![enter image description here](https://i.sstatic.net/Tt783LJj.png)](https://i.sstatic.net/Tt783LJj.png) This was at the start of training. However, as training progressed, the generator learnt from this gradient and was about to make better images when the gradient just started getting worse: [![enter image description here](https://i.sstatic.net/gzO3paIz.png)](htt
Comments.UserDisplayName: hidden_machine
Tags: <neural-networks><generative-adversarial-networks><performance><learning-rate>
Title: GAN performance starts to get worse as training continues
UserDisplayName: user17361867
ThreadText: # GAN performance starts to get worse as training continues
  
  (Asked by: username_0 on 2021-12-30)
  
  I'm currently trying to train a GAN to recreate similar images from a dataset. The dataset is using the Eiffel Tower Pictures from Googles [Quick Draw dataset](https://github.com/googlecreativelab/quickdraw-dataset). The images aren't very large (only 12x12 pixels) and are all black and white.
  
  The performance increases at an expected rate initially however after a certain point the quality begins to go down despite the cost from both generator and discriminator networks seeming to stay at a steady value as expected.
  
  [![Good Quality Image](https://i.sstatic.net/rK1PL.png)](https://i.sstatic.net/rK1PL.png)
  [![Bad Quality Image](https://i.sstatic.net/ZD8BV.png)](https://i.sstatic.net/ZD8BV.png)
  [![Worst Quality Image](https://i.sstatic.net/LzYmI.png)](https://i.sstatic.net/LzYmI.png)
  
  I've tried changing the learning rate and other hyper parameters however they all end up with the same result of the network eventually getting worse.
  
  My learning rate is currently 0.01 which I'm aware is quite high compared to what other people are using but anything lower takes too long to train and the results don't seem any better even if I do give it long enough.
  
  Any pointers on what could be causing this or the specific name of this problem if it's a common issue with GANs would be appreciated.
  
  Thanks
  
  **Tags:** neural-networks, generative-adversarial-networks, performance, learning-rate
  
  > **Comments:**
  > **username_1**: Which adversarial loss are you using? Also, is the drop in quality constant in time or do the images get worse/better/worse/better? It might be that the high learning rate push the weights out of the global minima after reaching it, in which case a scheduler or dynamic decrease of learning rate would be enough to solve the problem.  
  > **username_0**: Not quite sure what you mean by adversarial loss, but if you mean the loss function used by the generator and discriminator, it's the mean square error/quadratic loss function. The drop in quality seems to follow the pattern that pixels in the images produced later on get gradually blacker until all pixels are either completely black or white instead of also grey which produces worse images and doesn't get better. I'll try using a scheduler for the learning rate although the window of cycles in which good quality images are produced is quite slim so I'm not so sure of this working.  
  > **username_2**: were you ever able to get it working? I am having the same issue where the generated images get better after a while, but just keep getting worse and worse. I monitored the gradient coming into the generator for the picture of "3":[![enter image description here](https://i.sstatic.net/Tt783LJj.png)](https://i.sstatic.net/Tt783LJj.png) This was at the start of training. However, as training progressed, the generator learnt from this gradient and was about to make better images when the gradient just started getting worse: [![enter image description here](https://i.sstatic.net/gzO3paIz.png)](htt  
  
  ---
  
  ## 0 answers
  
  