Answers.Body: <p>Lets start with question <strong>1)</strong>  <em>how does JS-divergence handles zeros?</em>  </p>
  
  <p>by definition:<br>
  <span class="math-container">\begin{align}
  D_{JS}(p||q) &amp;= \frac{1}{2}[D_{KL}(p||\frac{p+q}{2}) + D_{KL}(q||\frac{p+q}{2})] \\
  &amp;= \frac{1}{2}\sum_{x\in\Omega} [p(x)log(\frac{2 p(x)}{p(x)+q(x)}) + q(x)log(\frac{2 q(x)}{p(x)+q(x)})]
  \end{align}</span>
  Where <span class="math-container">$\Omega$</span> is the union of the domains of <span class="math-container">$p$</span> and <span class="math-container">$q$</span>. Now lets assume one distribution is zero where the other is not (without loss of generality due to symmetry we can just say <span class="math-container">$p(x_i) = 0$</span> and <span class="math-container">$q(x_i) \neq 0$</span>. We then get for that term in the sum<br>
  <span class="math-container">$$\frac{1}{2}q(x_i)log(\frac{2q(x_i)}{q(x_i)}) = q(x_i)\frac{log(2)}{2}$$</span><br>
  Which isn't undefined as it would be the KL case.  </p>
  
  <p>Now onto <strong>2)</strong> <em>In GANS why does JS divergence produce better results than KL</em></p>
  
  <p>The asymmetry of KL divergence places an unfair advantage to one distribution over the other where in this case, its not ideal to consider it this way from an optimization perspective. Additionally KL divergences inability to handle non-overlapped distributions is crushing given that these are approximated through sampling schemes, and therefore there are no guarantees. JS solves both those issues and leads to a smoother manifold which is why its generally preferred. A good resource is <a href="https://arxiv.org/pdf/1511.05101.pdf" rel="nofollow noreferrer">this paper</a> where they go more in detail investigating this.    </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-12T00:58:24.067Z
Answers.LastActivityDate: 2019-11-12T00:58:24.067Z
Answers.UserDisplayName: mshlis
Body: <p>I have read articles on how Jensen-Shannon divergence is preferred over Kullback-Leibler in measuring how good a distribution mapping is learned in a generative network because of the fact that JS-divergence better measures distribution similarity when there are zero values in either distribution.</p>
  <p>I am unable to understand how the mathematical formulation of JS-divergence would take care of this and also what advantage it particularly holds qualitatively apart from this edge case.</p>
  <p>Could anyone explain or link me to an explanation that could answer this satisfactorily?</p>
  
Tags: <objective-functions><generative-adversarial-networks><probability-distribution><kl-divergence><jensen-shannon-divergence>
Title: Why is the Jensen-Shannon divergence preferred over the KL divergence in measuring the performance of a generative network?
UserDisplayName: ashenoy
ThreadText: # Why is the Jensen-Shannon divergence preferred over the KL divergence in measuring the performance of a generative network?
  
  (Asked by: username_0 on 2019-11-11)
  
  I have read articles on how Jensen-Shannon divergence is preferred over Kullback-Leibler in measuring how good a distribution mapping is learned in a generative network because of the fact that JS-divergence better measures distribution similarity when there are zero values in either distribution.
  
  I am unable to understand how the mathematical formulation of JS-divergence would take care of this and also what advantage it particularly holds qualitatively apart from this edge case.
  
  Could anyone explain or link me to an explanation that could answer this satisfactorily?
  
  **Tags:** objective-functions, generative-adversarial-networks, probability-distribution, kl-divergence, jensen-shannon-divergence
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-11-12. Score: 4 | Accepted answer
  
  Lets start with question **1)** *how does JS-divergence handles zeros?*
  
  by definition:  
  \begin{align}
  D\_{JS}(p||q) &= \frac{1}{2}[D\_{KL}(p||\frac{p+q}{2}) + D\_{KL}(q||\frac{p+q}{2})] \\
  &= \frac{1}{2}\sum\_{x\in\Omega} [p(x)log(\frac{2 p(x)}{p(x)+q(x)}) + q(x)log(\frac{2 q(x)}{p(x)+q(x)})]
  \end{align}
  Where $\Omega$ is the union of the domains of $p$ and $q$. Now lets assume one distribution is zero where the other is not (without loss of generality due to symmetry we can just say $p(x\_i) = 0$ and $q(x\_i) \neq 0$. We then get for that term in the sum  
  $$\frac{1}{2}q(x\_i)log(\frac{2q(x\_i)}{q(x\_i)}) = q(x\_i)\frac{log(2)}{2}$$  
  Which isn't undefined as it would be the KL case.
  
  Now onto **2)** *In GANS why does JS divergence produce better results than KL*
  
  The asymmetry of KL divergence places an unfair advantage to one distribution over the other where in this case, its not ideal to consider it this way from an optimization perspective. Additionally KL divergences inability to handle non-overlapped distributions is crushing given that these are approximated through sampling schemes, and therefore there are no guarantees. JS solves both those issues and leads to a smoother manifold which is why its generally preferred. A good resource is [this paper](https://arxiv.org/pdf/1511.05101.pdf) where they go more in detail investigating this.
  
  
  ---
  
  