Hello,
Im trying to use DistributedDataParallel to train a model in a cluster with nodes that each has 2 GPUs K80 and in total Im using 4 GPUs. Im based on this example for the distributed training: https://github.com/pytorch/examples/blob/master/imagenet/main.py 25.
My problem is that possibly it gets into a deadlock as it never terminates and never returns an error message, the terminal is totally freezing. I ve seen in the documentation of PyTorch the following:
If you plan on using this module with a nccl backend or a gloo
backend (that uses Infiniband), together with a DataLoader that uses
multiple workers, please change the multiprocessing start method to
forkserver (Python 3 only) or spawn. Unfortunately
Gloo (that uses Infiniband) and NCCL2 are not fork safe, and you will
likely experience deadlocks if you don’t change this setting.
and indeed im using gloo backend since DistributedDataParallel supports only gloo. But I tried with a single workers and I still have the same problem.
Also I ve seen in this topic Distributed data parallel freezes without error message 81 that PyTorch developers are currently working in the NCCL2 problem.
Does anybody know if currently there is any solution to the problem? Is there any way to avoid the deadlock and make the distributed training to work? Thank you!