I compile PyTorch v1.9.0 with CUDA 11.0 and NCCL 2.10.3

NCCL 2.10.3 Upgrade
https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub && add-apt-repository “deb Index of /compute/cuda/repos/ubuntu1804/x86_64 1 /” && apt update && apt install -y --allow-change-held-packages libnccl2=2.10.3-1+cuda11.0 libnccl-dev=2.10.3-1+cuda11.0

compile PyTorch v1.9.0 from source
git clone GitHub - pytorch/pytorch: Tensors and Dynamic neural networks in Python with strong GPU acceleration && cd pytorch && git checkout v1.9.0 && git submodule sync && git submodule update --init --recursive && sudo USE_SYSTEM_NCCL=1 TORCH_CUDA_ARCH_LIST=“6.0 6.1 7.0 7.5 8.0” python3 setup.py install

compile AWS-OFI-NCCL to support AWS EFA (fast cross machine communication in AWS)

  
      

      GitHub
  

  
    

GitHub - aws/aws-ofi-nccl: This is a plugin which lets EC2 developers use... 2

  This is a plugin which lets EC2 developers use libfabric as network provider while running NCCL applications. - GitHub - aws/aws-ofi-nccl: This is a plugin which lets EC2 developers use libfabric a...


  

  
    
    
  

  


git clone https://github.com/aws/aws-ofi-nccl.git $HOME/aws-ofi-nccl \
	     && cd $HOME/aws-ofi-nccl \
	     && git checkout aws  \
	     && ./autogen.sh \
	     && ./configure --prefix=$HOME/aws-ofi-nccl/install \
	        --with-libfabric=/opt/amazon/efa/ \
	        --with-cuda=/usr/local/cuda \
	        --with-nccl=/tmp/pytorch/build/nccl \
	        --with-mpi=/opt/amazon/openmpi/ \
	     && make && make install
	     && make -j$(nproc) && make install


Got the following error when using all_to_all APIs:
10.4.22.101:   File "/usr/local/lib/python3.6/dist-packages/m5_transformers/models/switch_transformers/switch_transformer_layers.py", line 310, in forward
10.4.22.101:     expert_inputs = self.Shuffle(torch.cat(route_inputs))
10.4.22.101:   File "/usr/local/lib/python3.6/dist-packages/m5_transformers/models/switch_transformers/switch_transformer_layers.py", line 508, in Shuffle
10.4.22.101:     return _Shuffle.apply(x)
10.4.22.101:   File "/usr/local/lib/python3.6/dist-packages/m5_transformers/models/switch_transformers/switch_transformer_layers.py", line 398, in forward
10.4.22.101:     return _shuffle(input_)
10.4.22.101:   File "/usr/local/lib/python3.6/dist-packages/m5_transformers/models/switch_transformers/switch_transformer_layers.py", line 202, in _shuffle
10.4.22.101:     output_tensor_list, input_tensor_list, mpu.get_data_parallel_group()
10.4.22.101:   File "/usr/local/lib/python3.6/dist-packages/torch/distributed/distributed_c10d.py", line 2478, in all_to_all
10.4.22.101:     work = group.alltoall(output_tensor_list, input_tensor_list, opts)
10.4.22.101: RuntimeError: NCCL error in: /tmp/pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:38, unhandled system error, NCCL version 21.0.3
10.4.22.101: ncclSystemError: System call (socket, malloc, munmap, etc) failed.