In terms of devices I have 2 GeForce RTX 2080 Ti

In terms of “reproducible” the loss always ends up at nan but it will not occur at the same epoch. Sometimes it will be epoch 1 and others it will be at epoch 9. I’ve even tried adding random seeds, but it does not help.
output from using torch.autograd.set_detect_anomaly(True)
Traceback (most recent call last):
  File "train.py", line 71, in <module>
    loss.backward()
  File "/home/anaconda3/envs/torch_train_env/lib/python3.7/site-packages/torch/tensor.py", line 198, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/anaconda3/envs/torch_train_env/lib/python3.7/site-packages/torch/autograd/__init__.py", line 100, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: Function 'LogSoftmaxBackward' returned nan values in its 0th output.