If you’re trying to average gradients across processes do this instead (it’s going to be faster than reduce + broadcast):
grad = param.grad
avg_grad = grad / num_processes
dist.all_reduce(avg_grad)

Also, you might want to use torch.nn.parallel.DistributedDataParallel instead. It will automatically optimize the transfers to give you better performance