Answers.Body: <p>By resampling from the generator after each update to the discriminator, you ensure that the generator is adapting to the evolving discriminator <em>and</em> producing diverse and realistic fake samples in response to the current state of the discriminator. If you sampled only once in one iteration, the generator would be updated based on the same set of fake samples used in the previous discriminator update, leading to potential mode collapse where the generator's training in the same iteration after discriminator's training becomes overly focused on a subset of possible outputs and fails to produce diverse and realistic samples and losing its adversarial training nature, essentially just a slowly improved actor always reacting to an unfriendly critic.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-12T07:45:00.250Z
Answers.Text: Yeah, I arrived at a similar conclusion. The discriminator would have an 'unfair' advantage as it would have experienced the samples already when evaluating the generator loss. I do still wonder how much of an issue it is practically, as I do not have a good intuition about how much of an advantage a single exposure is, especially at low learning rates. But I guess that's a question that no one really tested given that it conceptually doesn't make sense to omit the second sampling anyways.
Answers.UserDisplayName: John Smith
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-12T21:14:00.053Z
Answers.Text: GAN's generator is always deterministic to directly output synthetic data capturing only their rough distribution without (conditional) probabilities like VAEs, so intuitively mode collapse is a much more serious problem for GAN's generator and a second random noise as exploration in the same adversarial iteration definitely helps.
Answers.UserDisplayName: cinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-11T06:42:06.397Z
Answers.LastActivityDate: 2024-03-11T06:47:31.880Z
Answers.LastEditDate: 2024-03-11T06:47:31.880Z
Answers.UserDisplayName: cinch
Body: <p>Most basic GAN algorithm definitions I found go like this:</p>
  <ol>
  <li>Generate</li>
  <li>Train discriminator</li>
  <li>Generate</li>
  <li>Train generator</li>
  </ol>
  <p>Like this one: <a href="https://miro.medium.com/v2/resize:fit:720/format:webp/1*hlFyF-klXQunFpmoeA89jQ.png" rel="nofollow noreferrer">GAN pseudocode</a></p>
  <p>If I'm not misunderstanding, we sample twice from the same generator in a single iteration due to the fakes being generated before the weights are updated for the generator. Can you tell me why we need to sample twice?</p>
  
Tags: <generative-adversarial-networks>
Title: Why do you sample twice from the generator during a GAN training step?
UserDisplayName: John Smith
ThreadText: # Why do you sample twice from the generator during a GAN training step?
  
  (Asked by: username_0 on 2024-03-11)
  
  Most basic GAN algorithm definitions I found go like this:
  
  1. Generate
  2. Train discriminator
  3. Generate
  4. Train generator
  
  Like this one: [GAN pseudocode](https://miro.medium.com/v2/resize:fit:720/format:webp/1*hlFyF-klXQunFpmoeA89jQ.png)
  
  If I'm not misunderstanding, we sample twice from the same generator in a single iteration due to the fakes being generated before the weights are updated for the generator. Can you tell me why we need to sample twice?
  
  **Tags:** generative-adversarial-networks
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-03-11. Score: 1 | Accepted answer
  
  By resampling from the generator after each update to the discriminator, you ensure that the generator is adapting to the evolving discriminator *and* producing diverse and realistic fake samples in response to the current state of the discriminator. If you sampled only once in one iteration, the generator would be updated based on the same set of fake samples used in the previous discriminator update, leading to potential mode collapse where the generator's training in the same iteration after discriminator's training becomes overly focused on a subset of possible outputs and fails to produce diverse and realistic samples and losing its adversarial training nature, essentially just a slowly improved actor always reacting to an unfriendly critic.
  
  > **Comments:**
  > **username_0**: Yeah, I arrived at a similar conclusion. The discriminator would have an 'unfair' advantage as it would have experienced the samples already when evaluating the generator loss. I do still wonder how much of an issue it is practically, as I do not have a good intuition about how much of an advantage a single exposure is, especially at low learning rates. But I guess that's a question that no one really tested given that it conceptually doesn't make sense to omit the second sampling anyways.  
  > **username_1**: GAN's generator is always deterministic to directly output synthetic data capturing only their rough distribution without (conditional) probabilities like VAEs, so intuitively mode collapse is a much more serious problem for GAN's generator and a second random noise as exploration in the same adversarial iteration definitely helps.  
  
  ---
  
  