I am presently using the multigpu code developed using dist.all_reduce() function as in http://pytorch.org/tutorials/intermediate/dist_tuto.html  (the reason being that Dataparallel has the issue of “tensors are on different GPUs” and I have spent a lot of time to rectify it, but it sill doesn’t work). Do you suggest that dist.all_reduce is worse than nn.DataParallel for 8 GPUs?