What do you mean by “training doesn’t stop”?
Also, how do you run allreduce in asynchronous mode? The synchronization done by torch.nn.parallel.DistributedDataParallel is done implicitly, when you make autograd compute gradients for your model parameters. It doesn’t return until all the allreduce calls have finished (or in the case of CUDA tensors, until all the NCCL allreduce kernels have been queued).