Same problem here, with Pytorch v1.0.1 and NCCL backend. I am running one of the distributed examples from Ignite: https://github.com/pytorch/ignite/blob/master/examples/mnist/mnist_dist.py 14
Mine freezes on the line:
model = DistributedDataParallel(model, [args.gpu])
Where model is an instance of LeNet5 and args.gpu is 0.