I’m experiencing an intermittent issue with my GeForce 3090. The issue was occurring with nightly before 1.7 was released, and with the 1.7 version. It was also occurring with the nvidia driver 455.23 and after updating to 455.32.
For the release version, after installing 455.32, I installed PyTorch with:
conda install pytorch torchvision torchaudio cudatoolkit=11.0 -c pytorch

Sometimes the python process hangs after say the ten thousandth iteration or the 5 millionth. The issue occurs regardless the network that I am training (GAN, CNN, etc).
I don’t know that this is a PyTorch issue, but I am hopeful for suggestions on how to investigate what might be causing the hang so that I may report the issue to the appropriately. Any suggestions would be appreciated.