I found NCCL1 has a lower tensor size limit, and prepared this patch https://github.com/pytorch/pytorch/pull/11466 8 . However, this can really only happen if you have a huge tensor (not some tensors that adds up to be huge). Is this the case with your code?