Answers.Body: <p>When you one-hot-encode your labels with <span class="math-container">$p_i \in \{0,1\}$</span> you get <span class="math-container">$p_i = 0$</span> iff <span class="math-container">$i$</span> is not correct and, equivalently, <span class="math-container">$p_i =1$</span> iff <span class="math-container">$i$</span> is correct. </p>
  
  <p>Hence, <span class="math-container">$p_i \log(q_i) = 0 \log(q_i) = 0 $</span> for all classes except the "truth" and <span class="math-container">$p_i \log(q_i) = 1 \log(q_i) = \log(q_i) $</span> for the correct prediction. </p>
  
  <p>Therefore, your loss reduces to:
  <span class="math-container">$$
  H(p,q) = - \sum p_i \log(q_i) = - \log(q_{truth})
  $$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-15T08:23:03.580Z
Answers.Text: @DuttaA have a look at the last part of the question: "that I think it only addresses the $\log(q_i)$ part in the first formula. So is that means Pytorch using different Cross entropy formula?". Reads to me that it actually is exactly about this. .
Answers.UserDisplayName: Jonathan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-15T08:35:11.623Z
Answers.Text: OMG! yes, this is what I meant, I accidentally mix language modeling cross-entropy (that have continuous values of $p_i$) and PyTorch `nn.CrossEntropyLoss`. I forgot in classification $p_i$ will be 0 or 1 so the PyTorch function is valid, but of course, I can't use it directly for Language Modeling case. Thank You!
Answers.UserDisplayName: malioboro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-15T13:17:16.983Z
Answers.Text: @Sammy This answer partially answers the question, but I think you should also explain the function inside the log (in the PyTorch version). What function is that? Note that I know what function that is, but, for completeness, I think you should also explain it. Maybe you should also refer to the documentation. Afterward, I will upvote your answer ;)
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-15T07:52:55.613Z
Answers.LastActivityDate: 2020-01-15T07:52:55.613Z
Answers.UserDisplayName: Jonathan
Body: <p>In my understanding, the formula to calculate the cross-entropy is</p>
  
  <p><span class="math-container">$$
  H(p,q) = - \sum p_i \log(q_i)
  $$</span></p>
  
  <p>But in PyTorch <code>nn.CrossEntropyLoss</code> is calculated using this formula:</p>
  
  <p><span class="math-container">$$
  loss = -\log\left( \frac{\exp(x[class])}{\sum_j \exp(x_j)} \right)
  $$</span></p>
  
  <p>that I think it only addresses the <span class="math-container">$\log(q_i)$</span> part in the first formula. </p>
  
  <p>Why does PyTorch use a different formula for the cross-entropy?</p>
  
Tags: <neural-networks><deep-learning><objective-functions><pytorch><cross-entropy>
Title: Why does PyTorch use a different formula for the cross-entropy?
UserDisplayName: malioboro
ThreadText: # Why does PyTorch use a different formula for the cross-entropy?
  
  (Asked by: username_0 on 2020-01-15)
  
  In my understanding, the formula to calculate the cross-entropy is
  
  $$
  H(p,q) = - \sum p\_i \log(q\_i)
  $$
  
  But in PyTorch `nn.CrossEntropyLoss` is calculated using this formula:
  
  $$
  loss = -\log\left( \frac{\exp(x[class])}{\sum\_j \exp(x\_j)} \right)
  $$
  
  that I think it only addresses the $\log(q\_i)$ part in the first formula.
  
  Why does PyTorch use a different formula for the cross-entropy?
  
  **Tags:** neural-networks, deep-learning, objective-functions, pytorch, cross-entropy
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-01-15. Score: 1 | Accepted answer
  
  When you one-hot-encode your labels with $p\_i \in \{0,1\}$ you get $p\_i = 0$ iff $i$ is not correct and, equivalently, $p\_i =1$ iff $i$ is correct.
  
  Hence, $p\_i \log(q\_i) = 0 \log(q\_i) = 0 $ for all classes except the "truth" and $p\_i \log(q\_i) = 1 \log(q\_i) = \log(q\_i) $ for the correct prediction.
  
  Therefore, your loss reduces to:
  $$
  H(p,q) = - \sum p\_i \log(q\_i) = - \log(q\_{truth})
  $$
  
  > **Comments:**
  > **username_1**: @DuttaA have a look at the last part of the question: "that I think it only addresses the $\log(q_i)$ part in the first formula. So is that means Pytorch using different Cross entropy formula?". Reads to me that it actually is exactly about this. .  
  > **username_0**: OMG! yes, this is what I meant, I accidentally mix language modeling cross-entropy (that have continuous values of $p_i$) and PyTorch `nn.CrossEntropyLoss`. I forgot in classification $p_i$ will be 0 or 1 so the PyTorch function is valid, but of course, I can't use it directly for Language Modeling case. Thank You!  
  > **username_2**: @Sammy This answer partially answers the question, but I think you should also explain the function inside the log (in the PyTorch version). What function is that? Note that I know what function that is, but, for completeness, I think you should also explain it. Maybe you should also refer to the documentation. Afterward, I will upvote your answer ;)  
  
  ---
  
  