Is there a specific reason why you are using a custom loss function rather than the built in cross entropy loss? CrossEntropyLoss — PyTorch 1.9.0 documentation
Assuming there is a one-hot encoding being used for the targets, I’m not sure that this is a correct formulation of the loss function. After all, what happens when the output of the model is the same across all output classes (only the output of the correct class appears to be used)? However, this would not necessarily explain the reason the model weights are small and NaN out. You might want to also check your learning rate schedule and if decreasing the learning rate changes this behavior.