Your are right in that there is a regression when we broadcast the tensors and I’m working on a fix. However, it should really only happen when you don’t have NCCL, and we ship with NCCL1 in binary… So I am a bit confused. Could you tell me how you installed PyTorch?