Using round_robin_process_group with NCCL is not currently recommended. Check out the warning under: Distributed communication package - torch.distributed — PyTorch master documentation 15 :


Using multiple process groups with the NCCL backend concurrently is not safe and the user should perform explicit synchronization in their application to ensure only one process group is used at a time. This means collectives from one process group should have completed execution on the device (not just enqueued since CUDA execution is async) before collectives from another process group are enqueued. See Using multiple NCCL communicators concurrently 5 for more details.