Answers.Body: <p>I don't have a definite answer, but only a suspicion/idea:</p>
  <p>Looking at Figure 1 from the <a href="https://arxiv.org/pdf/1701.07875.pdf" rel="nofollow noreferrer">WGAN paper</a>, we clear see that the JS divergence on the right is not continuous at <span class="math-container">$0$</span>, hence not differentiable at <span class="math-container">$0$</span>. However, the EM plot on the left is continuous also at <span class="math-container">$0$</span>. You could now argue that we have a kink there, so it should not be differentiable there either, but they might have a different notion of differentiability, I am honestly not sure about it right now.  <a href="https://i.sstatic.net/0gXDg.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/0gXDg.png" alt="enter image description here" /></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-25T21:34:58.450Z
Answers.Text: There is a fairly straightforward way of doing optimization of piecewise linear functions that would deal fine with the left hand side, the non-differentiability at the 'kink' is not an issue, see e.g. [here](http://www.seas.ucla.edu/~vandenbe/ee236a/lectures/pwl.pdf)
Answers.UserDisplayName: Stiofán Fordham
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-27T11:17:16.150Z
Answers.LastActivityDate: 2020-12-27T11:17:16.150Z
Answers.UserDisplayName: Anonymous5638
Body: <p><a href="https://lilianweng.github.io/lil-log/2017/08/20/from-GAN-to-WGAN.html" rel="nofollow noreferrer">In this article</a> I am reading:</p>
  <blockquote>
  <p><span class="math-container">$D_{KL}$</span> gives us inifity when two distributions are disjoint. The value of <span class="math-container">$D_{JS}$</span> has sudden jump, not differentiable at <span class="math-container">$\theta=0$</span>. Only Wasserstein metric provides a smooth measure, which is super helpful for a stable learning process using gradient descents.</p>
  </blockquote>
  <p>Why is this important for a stable learning process? I have also the feeling this is also the reason for mode collapse in GANs, but I am not sure.</p>
  <p>The Wasserstein GAN paper also talks about it obviously, but I think I am missing a point. Does it say JS does not provide a usable gradient? What exactly does that mean?</p>
  
Tags: <neural-networks><generative-adversarial-networks><kl-divergence><wasserstein-metric><wasserstein-gan>
Title: What is the reason for mode collapse in GAN as opposed to WGAN?
UserDisplayName: craft
ThreadText: # What is the reason for mode collapse in GAN as opposed to WGAN?
  
  (Asked by: username_0 on 2020-07-25)
  
  [In this article](https://lilianweng.github.io/lil-log/2017/08/20/from-GAN-to-WGAN.html) I am reading:
  
  > $D\_{KL}$ gives us inifity when two distributions are disjoint. The value of $D\_{JS}$ has sudden jump, not differentiable at $\theta=0$. Only Wasserstein metric provides a smooth measure, which is super helpful for a stable learning process using gradient descents.
  
  Why is this important for a stable learning process? I have also the feeling this is also the reason for mode collapse in GANs, but I am not sure.
  
  The Wasserstein GAN paper also talks about it obviously, but I think I am missing a point. Does it say JS does not provide a usable gradient? What exactly does that mean?
  
  **Tags:** neural-networks, generative-adversarial-networks, kl-divergence, wasserstein-metric, wasserstein-gan
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-12-27. Score: 2
  
  I don't have a definite answer, but only a suspicion/idea:
  
  Looking at Figure 1 from the [WGAN paper](https://arxiv.org/pdf/1701.07875.pdf), we clear see that the JS divergence on the right is not continuous at $0$, hence not differentiable at $0$. However, the EM plot on the left is continuous also at $0$. You could now argue that we have a kink there, so it should not be differentiable there either, but they might have a different notion of differentiability, I am honestly not sure about it right now. [![enter image description here](https://i.sstatic.net/0gXDg.png)](https://i.sstatic.net/0gXDg.png)
  
  > **Comments:**
  > **username_2**: There is a fairly straightforward way of doing optimization of piecewise linear functions that would deal fine with the left hand side, the non-differentiability at the 'kink' is not an issue, see e.g. [here](http://www.seas.ucla.edu/~vandenbe/ee236a/lectures/pwl.pdf)  
  
  ---
  
  