we use NCCL Test (GitHub - NVIDIA/nccl-tests: NCCL Tests 4) and found the following bug when using alltoall():
Starting a DeepSpeed Training
+ cd /fsx/hchaoyan/m5/nccl-tests
++ which mpirun
+ /usr/local/mpi/bin/mpirun -allow-run-as-root --mca plm_rsh_no_tree_spawn 1 -x FI_PROVIDER=efa -x NCCL_SOCKET_IFNAME=eth -x FI_EFA_USE_DEVICE_RDMA=1 -x RDMAV_FORK_SAFE=1 -x LD_LIBRARY_PATH=/opt/nccl/build/lib:/usr/local/cuda/lib64:/opt/amazon/efa/lib64:/opt/amazon/openmpi/lib64:/opt/aws-ofi-nccl/lib:/usr/lib:/usr/local/lib:/usr/local/lib:/usr/local/mpi/lib:/usr/local/mpi/lib64:/usr/local/nvidia/lib:/usr/local/nvidia/lib64 -x NCCL_DEBUG=WARN -bind-to none -x NCCL_MIN_NCHANNELS=8 -x NCCL_ALGO=Ring -x OMP_NUM_THREADS=8 -x NCCL_NSOCKS_PERTHREAD=8 -x NCCL_SOCKET_NTHREADS=8 -n 16 -N 8 --mca pml '^cm' --hostfile /job/hostfile -mca btl tcp,self --mca btl_tcp_if_exclude lo,docker0 ./build/alltoall_perf -b 0.5G -e 2G -f 2 -g 1 -c 1 -n 10
Warning: Permanently added '[10.4.22.101]:2022' (RSA) to the list of known hosts.
# nThread 1 nGpus 1 minBytes 536870912 maxBytes 2147483648 step: 2(factor) warmup iters: 5 iters: 10 validation: 1 
#
# Using devices
#   Rank  0 Pid    311 on ip-10-4-2-84 device  0 [0x10] A100-SXM4-40GB
#   Rank  1 Pid    312 on ip-10-4-2-84 device  1 [0x10] A100-SXM4-40GB
#   Rank  2 Pid    313 on ip-10-4-2-84 device  2 [0x20] A100-SXM4-40GB
#   Rank  3 Pid    314 on ip-10-4-2-84 device  3 [0x20] A100-SXM4-40GB
#   Rank  4 Pid    315 on ip-10-4-2-84 device  4 [0x90] A100-SXM4-40GB
#   Rank  5 Pid    316 on ip-10-4-2-84 device  5 [0x90] A100-SXM4-40GB
#   Rank  6 Pid    319 on ip-10-4-2-84 device  6 [0xa0] A100-SXM4-40GB
#   Rank  7 Pid    321 on ip-10-4-2-84 device  7 [0xa0] A100-SXM4-40GB
#   Rank  8 Pid    286 on ip-10-4-22-101 device  0 [0x10] A100-SXM4-40GB
#   Rank  9 Pid    287 on ip-10-4-22-101 device  1 [0x10] A100-SXM4-40GB
#   Rank 10 Pid    288 on ip-10-4-22-101 device  2 [0x20] A100-SXM4-40GB
#   Rank 11 Pid    289 on ip-10-4-22-101 device  3 [0x20] A100-SXM4-40GB
#   Rank 12 Pid    290 on ip-10-4-22-101 device  4 [0x90] A100-SXM4-40GB
#   Rank 13 Pid    291 on ip-10-4-22-101 device  5 [0x90] A100-SXM4-40GB
#   Rank 14 Pid    292 on ip-10-4-22-101 device  6 [0xa0] A100-SXM4-40GB
#   Rank 15 Pid    296 on ip-10-4-22-101 device  7 [0xa0] A100-SXM4-40GB
NCCL version 2.9.9+cuda11.0
#
#                                                       out-of-place                       in-place          
#       size         count      type   redop     time   algbw   busbw  error     time   algbw   busbw  error
#        (B)    (elements)                       (us)  (GB/s)  (GB/s)            (us)  (GB/s)  (GB/s)       

ip-10-4-22-101:286:351 [0] transport/net_socket.cc:332 NCCL WARN Call to accept failed : Too many open files
ip-10-4-22-101: Test NCCL failure alltoall.cu:76 'unhandled system error'
 .. ip-10-4-22-101 pid 286: Test failure common.cu:505
 .. ip-10-4-22-101 pid 286: Test failure common.cu:694
 .. ip-10-4-22-101 pid 286: Test failure alltoall.cu:111
 .. ip-10-4-22-101 pid 286: Test failure common.cu:722
 .. ip-10-4-22-101 pid 286: Test failure common.cu:1083
 .. ip-10-4-22-101 pid 286: Test failure common.cu:925
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun.real detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[52734,1],8]
  Exit code:    3
--------------------------------------------------------------------------
+ '[' 3 -eq 0 ']'
+ log 'Writing exit code 1  to /tmp/batch-exit-code and shutting down supervisord'
+ echo 'mpi-run.sh - Writing exit code 1  to /tmp/batch-exit-code and shutting down supervisord'
mpi-run.sh - Writing exit code 1  to /tmp/batch-exit-code and shutting down supervisord
+ echo 1
++ cat /tmp/supervisord.pid
+ kill 7
+ exit 0