F.cross_entropy expects logits, so you shouldn’t apply softmax in the input, as internally F.log_softmax will be used.



 JBoRu:

what’s the difference between them


torch.log(torch.softmax()) should not be used, as it’s numerically less stable than torch.log_softmax.
For the dummy inputs of my previous post you’ll get:
print(torch.log_softmax(x, 1))
> tensor([[-1.0000e+11, -1.0986e+00, -1.0986e+00, -1.0986e+00]])

instead if the -inf output.



 JBoRu:

And this maskCrossEntropy is the chatbot tutorial of pytorch. And how to modify it?


Could you link to the line of code, which is using torch.log(torch.softmax()), please?