Answers.Body: <h3>Variance</h3>
  <p>What it seems to be taking place here is a form of overfitting. Specifically when using larger layers, the model may become too complex and start to fit the noise in the training data, rather than the underlying patterns. This can lead to poor generalization performance and, as seen here unstable behavior.</p>
  <h3>Other</h3>
  <p>Other complexities that may arise when using larger layers is diversity issues. Specifically, more complex GAN models may produce less diverse output from the generator, as the network is prone to get stuck in a particular mode of the data distribution and fail to explore other areas.</p>
  <p>To address this issue, regularisation techniques such as dropout or noise introduction can be deployed to encourage the generator to produce more diverse output. Additionally modifying the loss function can encourage the generator to explore different modes of the data distribution.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-20T14:36:45.927Z
Answers.LastActivityDate: 2023-04-20T14:36:45.927Z
Answers.UserDisplayName: hH1sG0n3
Body: <p>I am working with generative adversarial networks (GANs) and one of my aims at the moment is to reproduce samples in two dimensions that are distributed according to a circle (see animation). When using a GAN with small networks (<strong>3 layers with 50 neurons each</strong>), the results are more stable than with bigger layers (<strong>3 layers with 500 neurons each</strong>). <strong>All other hyperparameters are the same</strong> (see details of my implementation below).</p>
  <p>I am wondering if anyone has an explanation for why this is the case. I could obviously try to tune the other hyperparameters to get good performance but would be interested in knowing if someone has <strong>heuristics about what is needed to change whenever I change the size of the networks</strong>.</p>
  <p><a href="https://i.sstatic.net/pk3uO.gif" rel="nofollow noreferrer"><img src="https://i.sstatic.net/pk3uO.gif" alt="GAN with smaller layer size reproduces the original samples better" /></a></p>
  <hr />
  <h2>Network/Training parameters</h2>
  <p>I use PyTorch with the following settings for the GAN:</p>
  <p><strong>Networks:</strong></p>
  <ul>
  <li>Generator/Discriminator Architecture (all dense layers): 100-50-50-50-2 (small); 100-500-500-500-2 (big)</li>
  <li>Dropout: p=0.4 for generator (except last layer), p=0 for discriminator</li>
  <li>Activation functions: LeakyReLU (slope 0.1)</li>
  </ul>
  <p><strong>Training:</strong></p>
  <ul>
  <li>Optimizer: Adam</li>
  <li>Learning Rate: 1e-5 (for both networks)</li>
  <li>Beta1, Beta2: 0.9, 0.999</li>
  <li>Batch size: 50</li>
  </ul>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-26T09:16:18.617Z
Comments.Text: How many data points did you used? Maybe increasing dataset n times reduces this effect.
Comments.UserDisplayName: Enes
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-26T09:28:14.237Z
Comments.Text: I use 100k data points in total for training. With a batch size of 50, that's 2000 updates for each epoch, and I have 600 epochs. I think 100k is big enough so that increasing it shouldn't change too much.
Comments.UserDisplayName: Mafu
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-17T19:25:46.150Z
Comments.Text: Maybe the bigger network is perceiving more random noise and replicating it better. Just to be sure, I'd take it easy with the dropout and/or increase the batch size.
Comments.UserDisplayName: Andre Goulart
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-20T14:23:59.710Z
Comments.Text: Generally I think I agree with Andre above. Out of curiosity, have you tried increasing the number of layers instead of increasing the number of neurons in each layer?
Comments.UserDisplayName: hH1sG0n3
Tags: <neural-networks><machine-learning><deep-learning><generative-adversarial-networks><heuristics>
Title: Why is my GAN more unstable with bigger networks?
UserDisplayName: Mafu
ThreadText: # Why is my GAN more unstable with bigger networks?
  
  (Asked by: username_0 on 2021-01-26)
  
  I am working with generative adversarial networks (GANs) and one of my aims at the moment is to reproduce samples in two dimensions that are distributed according to a circle (see animation). When using a GAN with small networks (**3 layers with 50 neurons each**), the results are more stable than with bigger layers (**3 layers with 500 neurons each**). **All other hyperparameters are the same** (see details of my implementation below).
  
  I am wondering if anyone has an explanation for why this is the case. I could obviously try to tune the other hyperparameters to get good performance but would be interested in knowing if someone has **heuristics about what is needed to change whenever I change the size of the networks**.
  
  [![GAN with smaller layer size reproduces the original samples better](https://i.sstatic.net/pk3uO.gif)](https://i.sstatic.net/pk3uO.gif)
  
  ---
  
  ## Network/Training parameters
  
  I use PyTorch with the following settings for the GAN:
  
  **Networks:**
  
  * Generator/Discriminator Architecture (all dense layers): 100-50-50-50-2 (small); 100-500-500-500-2 (big)
  * Dropout: p=0.4 for generator (except last layer), p=0 for discriminator
  * Activation functions: LeakyReLU (slope 0.1)
  
  **Training:**
  
  * Optimizer: Adam
  * Learning Rate: 1e-5 (for both networks)
  * Beta1, Beta2: 0.9, 0.999
  * Batch size: 50
  
  **Tags:** neural-networks, machine-learning, deep-learning, generative-adversarial-networks, heuristics
  > **Comments:**
  > **username_1**: How many data points did you used? Maybe increasing dataset n times reduces this effect.  
  > **username_0**: I use 100k data points in total for training. With a batch size of 50, that's 2000 updates for each epoch, and I have 600 epochs. I think 100k is big enough so that increasing it shouldn't change too much.  
  > **username_2**: Maybe the bigger network is perceiving more random noise and replicating it better. Just to be sure, I'd take it easy with the dropout and/or increase the batch size.  
  > **username_3**: Generally I think I agree with Andre above. Out of curiosity, have you tried increasing the number of layers instead of increasing the number of neurons in each layer?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2023-04-20. Score: 1
  
  ### Variance
  
  What it seems to be taking place here is a form of overfitting. Specifically when using larger layers, the model may become too complex and start to fit the noise in the training data, rather than the underlying patterns. This can lead to poor generalization performance and, as seen here unstable behavior.
  
  ### Other
  
  Other complexities that may arise when using larger layers is diversity issues. Specifically, more complex GAN models may produce less diverse output from the generator, as the network is prone to get stuck in a particular mode of the data distribution and fail to explore other areas.
  
  To address this issue, regularisation techniques such as dropout or noise introduction can be deployed to encourage the generator to produce more diverse output. Additionally modifying the loss function can encourage the generator to explore different modes of the data distribution.
  
  
  ---
  
  