finally, we found the AWS EFA NCCL library doesn’t support v2.10.3 yet. It shows error:
10.4.2.84: ip-10-4-2-84:853:853 [6] find_ofi_provider:543 NCCL WARN NET/OFI Couldn't find any optimal provider
10.4.2.84: ip-10-4-2-84:848:848 [1] NCCL INFO NET/IB : No device found.
10.4.2.84: ip-10-4-2-84:848:848 [1] NCCL INFO NET/Socket : Using [0]eth0:10.4.2.84<0> [1]eth1:10.4.8.96<0> [2]eth2:10.4.28.87<0> [3]eth3:10.4.18.220<0>
10.4.2.84: ip-10-4-2-84:848:848 [1] NCCL INFO Using network Socket
10.4.2.84: ip-10-4-2-84:852:852 [5] NCCL INFO NET/IB : No device found.
10.4.2.84: ip-10-4-2-84:852:852 [5] NCCL INFO NET/Socket : Using [0]eth0:10.4.2.84<0> [1]eth1:10.4.8.96<0> [2]eth2:10.4.28.87<0> [3]eth3:10.4.18.220<0>
10.4.2.84: ip-10-4-2-84:852:852 [5] NCCL INFO Using network Socket
10.4.2.84: ip-10-4-2-84:851:851 [4] NCCL INFO NET/IB : No device found.
10.4.2.84: ip-10-4-2-84:851:851 [4] NCCL INFO NET/Socket : Using [0]eth0:10.4.2.84<0> [1]eth1:10.4.8.96<0> [2]eth2:10.4.28.87<0> [3]eth3:10.4.18.220<0>
10.4.2.84: ip-10-4-2-84:851:851 [4] NCCL INFO Using network Socket
10.4.2.84: ip-10-4-2-84:853:853 [6] NCCL INFO NET/IB : No device found.
10.4.2.84: ip-10-4-2-84:853:853 [6] NCCL INFO NET/Socket : Using [0]eth0:10.4.2.84<0> [1]eth1:10.4.8.96<0> [2]eth2:10.4.28.87<0> [3]eth3:10.4.18.220<0>
10.4.2.84: ip-10-4-2-84:853:853 [6] NCCL INFO Using network Socket
10.4.22.101: ip-10-4-22-101:743:743 [5] NCCL INFO Bootstrap : Using eth0:10.4.22.101<0>
10.4.22.101: ip-10-4-22-101:740:740 [2] NCCL INFO Bootstrap : Using eth0:10.4.22.101<0>