Answers.Body: <p>When it comes to a classification problem in machine learning, the cross-entropy and the KL divergence <strong>are equal</strong>.</p>
  <p>As already stated in the question, the general formula is this:</p>
  <p><span class="math-container">$$H(p, q) = H(p) + D_{KL}(p \parallel q),$$</span></p>
  <p>where <span class="math-container">$p$</span> is the &quot;true&quot;/target distribution and <span class="math-container">$q$</span> is an estimated distribution, <span class="math-container">$H(p, q)$</span> is the cross-entropy, <span class="math-container">$H(p)$</span> is the entropy and <span class="math-container">$D$</span> is the Kullback-Leibler divergence.</p>
  <p>Note that in machine learning, <span class="math-container">$p$</span> is a one-hot representation of the ground-truth class, i.e.,</p>
  <p><span class="math-container">$$p = [0,..., 1, ..., 0]$$</span></p>
  <p>which is basically a <a href="https://en.wikipedia.org/wiki/Dirac_delta_function" rel="noreferrer">delta-function</a> distribution. But the entropy of the delta function is zero, hence KL divergence simply equals the cross-entropy.</p>
  <p>In fact, even if <span class="math-container">$H(p)$</span> wasn't <span class="math-container">$0$</span> (e.g., soft labels), it is fixed and has no contribution to the gradient. In terms of optimization, it's safe to simply remove it and optimize the Kullback-Leibler divergence.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2017-10-04T13:04:52.790Z
Answers.LastActivityDate: 2021-10-13T11:34:31.637Z
Answers.LastEditDate: 2021-10-13T11:34:31.637Z
Answers.UserDisplayName: Maxim
Body: <p>The cross-entropy is identical to the KL divergence plus the entropy of the target distribution. The KL divergence equals zero when the two distributions are the same, which seems more intuitive to me than the entropy of the target distribution, which is what the cross-entropy is on a match.</p>
  <p>I'm not saying there's more information in one of the other except that a human view may find a zero more intuitive than a positive.
  Of course, one usually uses an evaluative method to really see how well classification occurs. But is the choice of the cross-entropy over the KL divergence historic?</p>
  
Tags: <machine-learning><classification><comparison><cross-entropy><kl-divergence>
Title: Why has the cross-entropy become the classification standard loss function and not Kullback-Leibler divergence?
UserDisplayName: Josh Albert
ThreadText: # Why has the cross-entropy become the classification standard loss function and not Kullback-Leibler divergence?
  
  (Asked by: username_0 on 2017-03-30)
  
  The cross-entropy is identical to the KL divergence plus the entropy of the target distribution. The KL divergence equals zero when the two distributions are the same, which seems more intuitive to me than the entropy of the target distribution, which is what the cross-entropy is on a match.
  
  I'm not saying there's more information in one of the other except that a human view may find a zero more intuitive than a positive.
  Of course, one usually uses an evaluative method to really see how well classification occurs. But is the choice of the cross-entropy over the KL divergence historic?
  
  **Tags:** machine-learning, classification, comparison, cross-entropy, kl-divergence
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2017-10-04. Score: 18 | Accepted answer
  
  When it comes to a classification problem in machine learning, the cross-entropy and the KL divergence **are equal**.
  
  As already stated in the question, the general formula is this:
  
  $$H(p, q) = H(p) + D\_{KL}(p \parallel q),$$
  
  where $p$ is the "true"/target distribution and $q$ is an estimated distribution, $H(p, q)$ is the cross-entropy, $H(p)$ is the entropy and $D$ is the Kullback-Leibler divergence.
  
  Note that in machine learning, $p$ is a one-hot representation of the ground-truth class, i.e.,
  
  $$p = [0,..., 1, ..., 0]$$
  
  which is basically a [delta-function](https://en.wikipedia.org/wiki/Dirac_delta_function) distribution. But the entropy of the delta function is zero, hence KL divergence simply equals the cross-entropy.
  
  In fact, even if $H(p)$ wasn't $0$ (e.g., soft labels), it is fixed and has no contribution to the gradient. In terms of optimization, it's safe to simply remove it and optimize the Kullback-Leibler divergence.
  
  
  ---
  
  