found the issue.

  
      github.com/pytorch/pytorch
  
  
    



Issue: distributed all_reduce deadlocks in v1.1 389


	
	opened by prafullasd
	on 2019-05-17
	
	
	closed by facebook-github-bot
	on 2019-05-26
	


🐛 Bug
I'm doing multi-node training (8 nodes, 8 gpu's each, NCCL backend) and am using DistributedDataParallel for syncing grads and distributed.all_reduce()...


 	module: cuda
 	module: distributed
 	triaged