<p>When you one-hot-encode your labels with <span class="math-container">$p_i \in \{0,1\}$</span> you get <span class="math-container">$p_i = 0$</span> iff <span class="math-container">$i$</span> is not correct and, equivalently, <span class="math-container">$p_i =1$</span> iff <span class="math-container">$i$</span> is correct. </p>

<p>Hence, <span class="math-container">$p_i \log(q_i) = 0 \log(q_i) = 0 $</span> for all classes except the "truth" and <span class="math-container">$p_i \log(q_i) = 1 \log(q_i) = \log(q_i) $</span> for the correct prediction. </p>

<p>Therefore, your loss reduces to:
<span class="math-container">$$
H(p,q) = - \sum p_i \log(q_i) = - \log(q_{truth})
$$</span></p>

CC BY-SA 4.0
2020-01-15T08:23:03.580Z
@DuttaA have a look at the last part of the question: "that I think it only addresses the $\log(q_i)$ part in the first formula. So is that means Pytorch using different Cross entropy formula?". Reads to me that it actually is exactly about this. .
Jonathan
CC BY-SA 4.0
2020-01-15T08:35:11.623Z
OMG! yes, this is what I meant, I accidentally mix language modeling cross-entropy (that have continuous values of $p_i$) and PyTorch `nn.CrossEntropyLoss`. I forgot in classification $p_i$ will be 0 or 1 so the PyTorch function is valid, but of course, I can't use it directly for Language Modeling case. Thank You!
malioboro
CC BY-SA 4.0
2020-01-15T13:17:16.983Z
@Sammy This answer partially answers the question, but I think you should also explain the function inside the log (in the PyTorch version). What function is that? Note that I know what function that is, but, for completeness, I think you should also explain it. Maybe you should also refer to the documentation. Afterward, I will upvote your answer ;)
nbro
CC BY-SA 4.0
2020-01-15T07:52:55.613Z
2020-01-15T07:52:55.613Z
Jonathan
<p>In my understanding, the formula to calculate the cross-entropy is</p>

<p><span class="math-container">$$
H(p,q) = - \sum p_i \log(q_i)
$$</span></p>

<p>But in PyTorch <code>nn.CrossEntropyLoss</code> is calculated using this formula:</p>

<p><span class="math-container">$$
loss = -\log\left( \frac{\exp(x[class])}{\sum_j \exp(x_j)} \right)
$$</span></p>

<p>that I think it only addresses the <span class="math-container">$\log(q_i)$</span> part in the first formula. </p>

<p>Why does PyTorch use a different formula for the cross-entropy?</p>

<neural-networks><deep-learning><objective-functions><pytorch><cross-entropy>
Why does PyTorch use a different formula for the cross-entropy?
malioboro
# Why does PyTorch use a different formula for the cross-entropy?

(Asked by: username_0 on 2020-01-15)

In my understanding, the formula to calculate the cross-entropy is

$$
H(p,q) = - \sum p\_i \log(q\_i)
$$

But in PyTorch `nn.CrossEntropyLoss` is calculated using this formula:

$$
loss = -\log\left( \frac{\exp(x[class])}{\sum\_j \exp(x\_j)} \right)
$$

that I think it only addresses the $\log(q\_i)$ part in the first formula.

Why does PyTorch use a different formula for the cross-entropy?

**Tags:** neural-networks, deep-learning, objective-functions, pytorch, cross-entropy


---

## 1 answers

### Answer by username_1 on 2020-01-15. Score: 1 | Accepted answer

When you one-hot-encode your labels with $p\_i \in \{0,1\}$ you get $p\_i = 0$ iff $i$ is not correct and, equivalently, $p\_i =1$ iff $i$ is correct.

Hence, $p\_i \log(q\_i) = 0 \log(q\_i) = 0 $ for all classes except the "truth" and $p\_i \log(q\_i) = 1 \log(q\_i) = \log(q\_i) $ for the correct prediction.

Therefore, your loss reduces to:
$$
H(p,q) = - \sum p\_i \log(q\_i) = - \log(q\_{truth})
$$

> **Comments:**
> **username_1**: @DuttaA have a look at the last part of the question: "that I think it only addresses the $\log(q_i)$ part in the first formula. So is that means Pytorch using different Cross entropy formula?". Reads to me that it actually is exactly about this. .  
> **username_0**: OMG! yes, this is what I meant, I accidentally mix language modeling cross-entropy (that have continuous values of $p_i$) and PyTorch `nn.CrossEntropyLoss`. I forgot in classification $p_i$ will be 0 or 1 so the PyTorch function is valid, but of course, I can't use it directly for Language Modeling case. Thank You!  
> **username_2**: @Sammy This answer partially answers the question, but I think you should also explain the function inside the log (in the PyTorch version). What function is that? Note that I know what function that is, but, for completeness, I think you should also explain it. Maybe you should also refer to the documentation. Afterward, I will upvote your answer ;)  

---

