I tried running a simple script that mimics your described code, but I am not able to reproduce the issue - this might be due to having different environments, etc. Do you mind creating a GitHub issue in the PyTorch repo and filling out the detailed environment description there (PT version, CUDA, NCCL version, etc.)? That way, we can try to reproduce this issue in an environment similar to yours. Thanks!