Hi,
I’ve recently gotten access to some A6000 GPUs. The machine has CUDA11.3 installed, and my environment has the latest PyTorch release (1.10.0) with the CUDA11.3 build: torch==1.10.0+cu113.
It seems like single GPU training works well, but as soon as I switch to DDP (initiated when torch.cuda.device_count() > 1 the training does not run.
Interestingly, the GPU memory get occupied by the appropriate amount, and GPU utilization is at 100%. However, the code does not progress from there on. No error messages - it just hangs.
Is this a known bug for A6000 GPUs? I’ve tested my DDP code on a wide range of GPUs (TitanXP, RTX 2080Ti, Quadro RTX 8000, etc) and haven’t faced issues. The code doesn’t even fail (just hangs), so I’m not sure how to start debugging this. Has anyone else faced similar issues?