Answers.Body: <p>I don't think he said that at all. Going back to the talk you'll see he mentions mode collapse comes from the naivete of using alternating gradient-based optimization steps because then <span class="math-container">$min_{\phi}max_{\theta}L(G_\phi, D_\theta)$</span> starts to look a lot like <span class="math-container">$max_{\theta}min_{\phi}L(G_\phi, D_\theta)$</span>.  </p>
  
  <p>This is problematic because in the latter case the generator has an obvious minimum of transforming all generated output into a single-mode that the discriminator has considered acceptable.  </p>
  
  <p>Since then a lot of work has been done to deal with this point of failure. Examples include <a href="https://arxiv.org/abs/1611.02163" rel="nofollow noreferrer">Unrolled GANs</a> (he mentions this one in the talk), where you essentially make the generator optimize what the discriminator <em>will</em> think <span class="math-container">$K$</span> steps in the future to ensure the ordering of the <span class="math-container">$min \ max$</span> game, and <a href="https://arxiv.org/abs/1701.07875" rel="nofollow noreferrer">Wasserstein GANs</a>, where you focus on a different metric that still has the same global minimum but allows for side by side training completely eliminating the ordering and failure mode, to begin with. On top of this, other work has been done as well, these are just two important examples.  </p>
  
  <p>Regarding how they fare against other generative models, like VAEs, there is no <em>one is better than the other</em>. The recent empirical success of GANs is why they are so popularly used, but we still see others being used in practice as well.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-12T16:23:17.270Z
Answers.LastActivityDate: 2019-11-13T14:45:25.077Z
Answers.LastEditDate: 2019-11-13T14:45:25.077Z
Answers.UserDisplayName: mshlis
Body: <p>Of my understanding mode-collapse is when there happen to be multiple classes in the dataset and the generative network converges to only one of these classes and generates images only within this class. On training the model more, the model converges to another class.</p>
  
  <p>In Goodfellows NeurIPS presentation he clearly addressed how training a generative network in an adversarial manner avoids mode-collapse. How exactly do GAN's avoid mode-collapse? and did previous works on generative networks not try to address this?</p>
  
  <p>Apart from the obvious superior performance (generally), is the fact that GAN's address mode-collapse make them far preferred over other ways of training a generative model?</p>
  
Tags: <neural-networks><deep-learning><training><generative-adversarial-networks><generative-model>
Title: How exactly does adversarial training help in handling mode-collapse in generative networks?
UserDisplayName: ashenoy
ThreadText: # How exactly does adversarial training help in handling mode-collapse in generative networks?
  
  (Asked by: username_0 on 2019-11-11)
  
  Of my understanding mode-collapse is when there happen to be multiple classes in the dataset and the generative network converges to only one of these classes and generates images only within this class. On training the model more, the model converges to another class.
  
  In Goodfellows NeurIPS presentation he clearly addressed how training a generative network in an adversarial manner avoids mode-collapse. How exactly do GAN's avoid mode-collapse? and did previous works on generative networks not try to address this?
  
  Apart from the obvious superior performance (generally), is the fact that GAN's address mode-collapse make them far preferred over other ways of training a generative model?
  
  **Tags:** neural-networks, deep-learning, training, generative-adversarial-networks, generative-model
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-11-12. Score: 1 | Accepted answer
  
  I don't think he said that at all. Going back to the talk you'll see he mentions mode collapse comes from the naivete of using alternating gradient-based optimization steps because then $min\_{\phi}max\_{\theta}L(G\_\phi, D\_\theta)$ starts to look a lot like $max\_{\theta}min\_{\phi}L(G\_\phi, D\_\theta)$.
  
  This is problematic because in the latter case the generator has an obvious minimum of transforming all generated output into a single-mode that the discriminator has considered acceptable.
  
  Since then a lot of work has been done to deal with this point of failure. Examples include [Unrolled GANs](https://arxiv.org/abs/1611.02163) (he mentions this one in the talk), where you essentially make the generator optimize what the discriminator *will* think $K$ steps in the future to ensure the ordering of the $min \ max$ game, and [Wasserstein GANs](https://arxiv.org/abs/1701.07875), where you focus on a different metric that still has the same global minimum but allows for side by side training completely eliminating the ordering and failure mode, to begin with. On top of this, other work has been done as well, these are just two important examples.
  
  Regarding how they fare against other generative models, like VAEs, there is no *one is better than the other*. The recent empirical success of GANs is why they are so popularly used, but we still see others being used in practice as well.
  
  
  ---
  
  