Answers.Body: <p>Your intuition doesn't coincide with my interpretation. This is how I see it:</p>
  <p>The generator might have some problem early on, for simplicity lets consider the example where the generator always outputs a black pixel in the top left corner. Then the discriminator learns to select on that particular feature, and the generator might take several update steps to resolve the issue, during which the discriminator might shift toward only selecting for the color of the first pixel instead of whether the whole image looks like a face.</p>
  <p>Because there is this interplay between the generator and discriminator and because they are constantly lacking behind each other they might start to oscillate. The generator might overcompensate for some random pixel which gave him away to the discriminator and then the discriminator might overcompensate by selecting for a single different pixel which the generator wasn't focused on.</p>
  <p>Of course this example is oversimplified by talking about the value of a single pixel, but you get the idea: the two networks are too focused on small problems.</p>
  <p>It's not about the discriminator getting too good, but about it getting too focused on one of the mistakes the generator is currently making.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-18T03:59:43.523Z
Answers.LastActivityDate: 2023-02-18T03:59:43.523Z
Answers.UserDisplayName: Tim Kuipers
Answers.Body: <blockquote>
  <p>What do they mean by &quot;the discriminator overshoots&quot;?</p>
  </blockquote>
  <p><strong>Mode collapse</strong> happens when the generator outputs the same or a set of few images that always look the same regardless of different input noise values. This provides a very easy way for the discriminator to learn that the generated images are fake.</p>
  <p>When this happens the discriminator loss is quite <em>low</em> (tending to zero) because it correctly classifies fake vs real images. The same occurs when training the generator instead as since it's trained by back-propagating the error of the discriminator, if the generated images are not able to fool the discriminator by reversing the class labels (due to images collapsed to few modes which are very easy to distinguish), its error is still low and so the <em>generator's gradient tends to vanish</em>, making impossible to let the generator improve and so to escape the local minimum. Indeed, when this happens it's useless to train further.</p>
  <p>An <strong>ideal training</strong> is characterized, instead, by none of the two networks to overcome the other, meaning that for some epochs the discriminator is fooled (say <span class="math-container">$60\%$</span> of the times) and so the generator improves, then the discriminator gets better at classifying fakes (but still misses many) still giving the chance for the generator to improve in the meanwhile. If at any point during training one network (especially the discriminator) strongly overcomes the other, then the training ends to a poor solution: a mode collapse for example, that can even happen when the quality of the generated samples is satisfactory - instead, if such process continues and converges, the generator has reached the best possible quality (while also having a wide variety), while the discriminator is not able to distinguish true vs generated images: so its predictions are around <span class="math-container">$0.5$</span> probability, that is the <em>Nash equilibrium</em> (i.e. optimal solution) of the <em>min-max game</em> between the generator and discriminator.</p>
  <blockquote>
  <p>What do they mean by &quot;the signal magnitudes escalate in both networks&quot;?</p>
  </blockquote>
  <p>I'm not sure about this point but I guess it refers to a diverging training scenario, in which the gradient of both networks starts to explode because the learned features of the generator (corresponding to predicted pixels) are out of control. In fact, to overcome this issue the authors decide to normalize the generator's feature of each pixel to unit norm. They call this trick <em>pixelwise normalization</em>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-26T19:25:56.547Z
Answers.LastActivityDate: 2023-04-26T19:25:56.547Z
Answers.UserDisplayName: Luca Anzalone
Answers.Body: <p>Yes, your intuition is correct. The effect of this problem is that the generator can no longer improve its output to marginally fool the discriminator - the discriminator isn't buying any of the generated output. In this case, the generator gets stuck in a local minimum and typically produces nonsense results.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-10T10:45:46.347Z
Answers.LastActivityDate: 2019-05-10T10:45:46.347Z
Answers.UserDisplayName: DrMcCleod
Answers.Body: <p>A discriminator overshooting may result in a dataset that has not been thoroughly clean and probably has too many identical feature, as a result there will be an early convergence from the discriminator as there is little variation. 
  The drawback from this is that the model will not bee able to generalize well.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-05T14:40:24.340Z
Answers.Text: In other words, the generator can no longer effective learn how to fool the discriminator because everything fails.
Answers.UserDisplayName: David Hoelzer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-06-12T09:06:03.607Z
Answers.LastActivityDate: 2018-06-12T09:06:03.607Z
Answers.UserDisplayName: Simbarashe Timothy Motsi
Body: <p>In the paper <a href="https://research.nvidia.com/sites/default/files/pubs/2017-10_Progressive-Growing-of/karras2018iclr-paper.pdf" rel="nofollow noreferrer">Progressive growing of gans for improved quality, stability, and variation</a> (ICLR, 2018) by Nvidia researchers, the authors write</p>
  <blockquote>
  <p>Furthermore, we observe that mode collapses traditionally
  plaguing GANs tend to happen very quickly, over the course of a dozen minibatches. Commonly
  they start when the discriminator overshoots, leading to exaggerated gradients, and an unhealthy
  competition follows where the signal magnitudes escalate in both networks. We propose a mechanism to stop the generator from participating in such escalation, overcoming the issue (Section 4.2)</p>
  </blockquote>
  <p>What do they mean by &quot;the discriminator overshoots&quot; and &quot;the signal magnitudes escalate in both networks&quot;?</p>
  <p>My current intuition is that the discriminator gets too good too soon, which causes the generator to spike and try to play catch up. That would be the unhealthy competition that they are talking about. Model collapse is the side effect where the generator has trouble playing catch up and decides to play it safe by generating slightly varied images to increase its accuracy. Is this way of interpreting the above paragraph correct?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-26T09:00:22.507Z
Comments.Text: Note that it's not "Model collapse" (as you write) but "mode collapse". This refers to the mode of a distribution (if I remember correctly).
Comments.UserDisplayName: nbro
Tags: <deep-learning><training><papers><generative-adversarial-networks><discriminator>
Title: Can some one help me understand this paragraph from Nvidia's progressive GAN paper?
UserDisplayName: Inkplay_
ThreadText: # Can some one help me understand this paragraph from Nvidia's progressive GAN paper?
  
  (Asked by: username_0 on 2018-06-06)
  
  In the paper [Progressive growing of gans for improved quality, stability, and variation](https://research.nvidia.com/sites/default/files/pubs/2017-10_Progressive-Growing-of/karras2018iclr-paper.pdf) (ICLR, 2018) by Nvidia researchers, the authors write
  
  > Furthermore, we observe that mode collapses traditionally
  > plaguing GANs tend to happen very quickly, over the course of a dozen minibatches. Commonly
  > they start when the discriminator overshoots, leading to exaggerated gradients, and an unhealthy
  > competition follows where the signal magnitudes escalate in both networks. We propose a mechanism to stop the generator from participating in such escalation, overcoming the issue (Section 4.2)
  
  What do they mean by "the discriminator overshoots" and "the signal magnitudes escalate in both networks"?
  
  My current intuition is that the discriminator gets too good too soon, which causes the generator to spike and try to play catch up. That would be the unhealthy competition that they are talking about. Model collapse is the side effect where the generator has trouble playing catch up and decides to play it safe by generating slightly varied images to increase its accuracy. Is this way of interpreting the above paragraph correct?
  
  **Tags:** deep-learning, training, papers, generative-adversarial-networks, discriminator
  > **Comments:**
  > **username_1**: Note that it's not "Model collapse" (as you write) but "mode collapse". This refers to the mode of a distribution (if I remember correctly).  
  
  ---
  
  ## 4 answers
  
  ### Answer by username_2 on 2023-02-18. Score: 1
  
  Your intuition doesn't coincide with my interpretation. This is how I see it:
  
  The generator might have some problem early on, for simplicity lets consider the example where the generator always outputs a black pixel in the top left corner. Then the discriminator learns to select on that particular feature, and the generator might take several update steps to resolve the issue, during which the discriminator might shift toward only selecting for the color of the first pixel instead of whether the whole image looks like a face.
  
  Because there is this interplay between the generator and discriminator and because they are constantly lacking behind each other they might start to oscillate. The generator might overcompensate for some random pixel which gave him away to the discriminator and then the discriminator might overcompensate by selecting for a single different pixel which the generator wasn't focused on.
  
  Of course this example is oversimplified by talking about the value of a single pixel, but you get the idea: the two networks are too focused on small problems.
  
  It's not about the discriminator getting too good, but about it getting too focused on one of the mistakes the generator is currently making.
  
  
  ---
  
  ### Answer by username_3 on 2023-04-26. Score: 0
  
  > What do they mean by "the discriminator overshoots"?
  
  **Mode collapse** happens when the generator outputs the same or a set of few images that always look the same regardless of different input noise values. This provides a very easy way for the discriminator to learn that the generated images are fake.
  
  When this happens the discriminator loss is quite *low* (tending to zero) because it correctly classifies fake vs real images. The same occurs when training the generator instead as since it's trained by back-propagating the error of the discriminator, if the generated images are not able to fool the discriminator by reversing the class labels (due to images collapsed to few modes which are very easy to distinguish), its error is still low and so the *generator's gradient tends to vanish*, making impossible to let the generator improve and so to escape the local minimum. Indeed, when this happens it's useless to train further.
  
  An **ideal training** is characterized, instead, by none of the two networks to overcome the other, meaning that for some epochs the discriminator is fooled (say $60\%$ of the times) and so the generator improves, then the discriminator gets better at classifying fakes (but still misses many) still giving the chance for the generator to improve in the meanwhile. If at any point during training one network (especially the discriminator) strongly overcomes the other, then the training ends to a poor solution: a mode collapse for example, that can even happen when the quality of the generated samples is satisfactory - instead, if such process continues and converges, the generator has reached the best possible quality (while also having a wide variety), while the discriminator is not able to distinguish true vs generated images: so its predictions are around $0.5$ probability, that is the *Nash equilibrium* (i.e. optimal solution) of the *min-max game* between the generator and discriminator.
  
  > What do they mean by "the signal magnitudes escalate in both networks"?
  
  I'm not sure about this point but I guess it refers to a diverging training scenario, in which the gradient of both networks starts to explode because the learned features of the generator (corresponding to predicted pixels) are out of control. In fact, to overcome this issue the authors decide to normalize the generator's feature of each pixel to unit norm. They call this trick *pixelwise normalization*.
  
  
  ---
  
  ### Answer by username_4 on 2019-05-10. Score: 0
  
  Yes, your intuition is correct. The effect of this problem is that the generator can no longer improve its output to marginally fool the discriminator - the discriminator isn't buying any of the generated output. In this case, the generator gets stuck in a local minimum and typically produces nonsense results.
  
  
  ---
  
  ### Answer by username_5 on 2018-06-12. Score: 0
  
  A discriminator overshooting may result in a dataset that has not been thoroughly clean and probably has too many identical feature, as a result there will be an early convergence from the discriminator as there is little variation.
  The drawback from this is that the model will not bee able to generalize well.
  
  > **Comments:**
  > **username_6**: In other words, the generator can no longer effective learn how to fool the discriminator because everything fails.  
  
  ---
  
  