I think you might not be considering the “confidence” of the predictions.
I.e. even if your model is predicting random classes, the loss would be much higher, if you scale the logits as:
input = torch.randn(3, 5) * 100

However, you should get approx. the same loss if you keep the logits close to the random output:
input = torch.zeros(3, 5) + torch.randn(3, 5) * 1e-3