@ptrblck, the problem is that this issue is not reproducible. It is totally random whether the training will face a deadlock or not. For example, I had three network architectures, and the day before yesterday, two of the three were suffering from NCCL deadlock while the other one was not. Yesterday when I tried it again, I had this issue with only one of the architectures and not with the other. I didn’t change a single line in the code.
I am using Pytorch 1.5.0 on Ubuntu 18.04. Also,
>>> torch.cuda.nccl.version()
2408

>>> torch.backends.cudnn.version()
7605

I am training my models on RTX 2080ti and in the multi-gpu set up, I have tried using 2,4 and 8 GPUs but the deadlock issue persists.
I am training image classification models and the training freezes on the line
images = images.cuda(non_blocking=True)

I have already experimented with non_blocking=False and I am sure that it is not the problem.