I tried NCCL v2.9.9 + PyTorch v1.9.0 + CUDA 11.0 + AWS-OFI-NCCL (aws branch), alltoall() operation still failed
10.4.22.101:   File "/usr/local/lib/python3.6/dist-packages/m5_transformers/models/switch_transformers/switch_transformer_layers.py", line 310, in forward
10.4.22.101:     expert_inputs = self.Shuffle(torch.cat(route_inputs))
10.4.22.101:   File "/usr/local/lib/python3.6/dist-packages/m5_transformers/models/switch_transformers/switch_transformer_layers.py", line 508, in Shuffle
10.4.22.101:     return _Shuffle.apply(x)
10.4.22.101:   File "/usr/local/lib/python3.6/dist-packages/m5_transformers/models/switch_transformers/switch_transformer_layers.py", line 398, in forward
10.4.22.101:     return _shuffle(input_)
10.4.22.101:   File "/usr/local/lib/python3.6/dist-packages/m5_transformers/models/switch_transformers/switch_transformer_layers.py", line 202, in _shuffle
10.4.22.101:     output_tensor_list, input_tensor_list, mpu.get_data_parallel_group()
10.4.22.101:   File "/usr/local/lib/python3.6/dist-packages/torch/distributed/distributed_c10d.py", line 2478, in all_to_all
10.4.22.101:     work = group.alltoall(output_tensor_list, input_tensor_list, opts)
10.4.22.101: RuntimeError: NCCL error in: /tmp/pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:38, unhandled system error, NCCL version 20.9.9