Hi hananelroe!



 hananelroe:

loss_fn = nn.CrossEntropyLoss()
...
    loss = loss_fn(prediction, expected)
    print(f"loss: {loss}")

expected: tensor([1])
prediction: tensor([[ 1.1202e+09, -2.1888e+09]], grad_fn=<AddmmBackward>)
loss: 3309037824.0
prediction: tensor([[-1.0482e+12,  1.1267e+12]], grad_fn=<AddmmBackward>)
loss: 0.0



First, with CrossEntropyLoss, you don’t expect your predicted
values (the input to CrossEnropyLoss) and the expected values
(the target for CrossEntropyLoss to be the same.  They are,
roughly speaking, encoded differently.
The input is a set of nClass logits, while the target is a single
integer class label (that runs from 0 to nClass - 1).
Now, to your specific case:
When you say expected is tensor ([1]), you are saying that you
have a batch of batch size 1 and that that single sample is in class
“1”.
When you say that your first value for prediction is
tensor ([[ 1.1202e+09, -2.1888e+09]]), you are saying that
you have a (very) large positive logit for the sample being in class
“0” and a (very) large negative logit for the sample being in class “1”.
When you (conceptually) convert the logits to probabilities, you are
predicting that the sample is in class “0” with a probability very
close to 1.0.  That is, you are making a highly certain incorrect
prediction and your loss is large.
Conversely, your second value for prediction,
tensor ([[-1.0482e+12,  1.1267e+12]]), is the other way around,
and corresponds to a prediction probability of very nearly 1.0 that
your sample is in class “1”.  You are making a highly certain correct
prediction so your loss is very close to 0.0.
Internal to CrossEntropyLoss is a softmax() function (actually
log_softmax()) that converts the input logits into probabilities.
Let’s see what we get when we apply softmax() to your values
for prediction:
>>> import torch
>>> torch.__version__
'1.9.0'
>>> prediction_1 = torch.tensor ([[ 1.1202e+09, -2.1888e+09]])
>>> prediction_2 = torch.tensor ([[-1.0482e+12,  1.1267e+12]])
>>> torch.nn.functional.softmax (prediction_1, dim = 1)
tensor([[1., 0.]])
>>> torch.nn.functional.softmax (prediction_2, dim = 1)
tensor([[0., 1.]])

In fact, your logits are so large that within the numerical range
of floating-point numbers, you get exactly 1.0 for the predicted
probability of being in class “0” and “1”, respectively, hence the
loss of exactly 0.0 for your second prediction.
Best.
K. Frank