If it’s possible, could you try an NGC PyTorch container: PyTorch | NVIDIA NGC? I’ve tried a few recent images (21.10, 21.08, 21.06) which should correlate to 1.10, 1.10, and 1.9 respectively but couldn’t reproduce the NaN issue. Testing on the same driver is a bit involved; for reference I was on 470.86.
Running the script on 2xA6000, the loss stayed between 8-9 after 80 epochs.
EDIT: I’ve also tried the 1.10+cu111 pip wheel and was unable to reproduce the issue.