we are aware of some of the multi-GPU cases on the extreme might have these issues.
We’re fixing these perf for the next release.
In the meanwhile, you can try using nccl2 instead of default nccl.
Download nccl2, and then do:
NCCL_ROOT_DIR=[where nccl2 is] python setup.py install

See: https://github.com/pytorch/pytorch#from-source 7