nn.CrossEntropyLoss is considers unsafe in float16 which is why it’s using float32 if you are using the mixed-precision training utilities via torch.cuda.amp.autocast as described here 4.
Since you are seeing invalid outputs, my best guess is that you are indeed hitting an overlow or other numerical issues.