Your note might be the solution to this issue. If you manually apply softmax on your output, note that nn.CrossEntropyLoss will internally apply F.log_softmax again on the model output, which might break or slow down your training significantly.
Could you try to change other hyperparameters (e.g. learning rate) so that you avoid computing Inf in your forward pass?