Hey Chaoyang, does Gloo backend work? It’s a known issue that NCCL backend might hang/crash when there are concurrent communications from different NCCL communicators (PyTorch creates one NCCL communicator per ProcessGroup instance) on the same device. But I am not sure if that’s relevant here. I assume the allToAll is called in model forward?