Cross Entropy in PyTorch is different from what I learnt (Not about logit input, but about the loss for every node) I know that the CrossEntropyLoss in Pytorch expects logits.  I also know that the reduction argument in CrossEntropyLoss is to reduce along the data sample's axis, if it is reduction=mean, that is to take $\frac{1}{m}\sum^m_{i=1}$. If reduction=sum, then it is $\sum^m_{i=1}$. If I use 'none', it will just give me a tensor list of loss of each data sample fed. So I am not asking about these two things.
My problem is that about the fact that what I have learnt about cross entropy is to calculate the loss for every output node. But in PyTorch, it only calculates for the class fed.
PyTorch's CrossEntropyLoss has a reduction argument, but it is to do mean or sum or none over the data samples axis.
Assume I am doing everything from scratch, that now I have a model, with 3 output nodes (data has 3 classes $C=3$), and I only pass one data sample $m=1$ to the model. I call the logits of the three output nodes $z_1,z_2,z_3$. If I code everything from scratch I am going to do softmax on them and I will obtain $\hat{y}_1,\hat{y}_2,\hat{y}_3$. Assume the label of this data sample is 0, so I convert it into one-hot label as $[y_1,y_2,y_3]=[1,0,0]$. After that I can use cross entropy to compute
$\sum^C_{k=1}[-y_k\log \hat{y}_k-(1-y_k)\log (1-\hat{y}_k)]$
So, for the final loss for gradient descent, i will sum all the 3 cross entropy loss for each node.
But in PyTorch, it will only calculate the one with the class 0 as the label for this data sample is 0
$-y_1\log \hat{y}_1-(1-y_1)\log (1-\hat{y}_1)$
and ignore others
Why is that?
To show it in code


The loss of a single sample is the sum of the $J$ nodes' predictions, so for each sample, the loss is a scalar value.
The expression $-\sum_j^J y_j \log p_j$ has only one nonzero term, because only one element of $y$ is nonzero. This is the correct loss for a classification problem; we can prove that this is correct by writing out the likelihood of the categorical distribution and taking the negative of its logarithm to get the cross-entropy. It's merely a special case that we can re-write the binary classification problem in terms of one probability $p$ and one binary label $y$ in the convenient form $-y \log p - (1-y)\log (1-p)$.
Let's write out a concrete example. We have predictions $[0.4,0.3,0.2]$ and label $[1,0,0]$. By inspection, we know that the loss is $-\log(0.4)$ because this is the only entry with a nonzero label.
