After digging into the original nn.CrossEntropyLoss implementation.
The problem was summing the loss on the batch instead of averaging it. It was surprising to see that I get such different results once summing since the batch size I am using is very small (one sentence which is ~30 words).
instead of:
loss.sum().backward()
use:
loss.mean().backward()