The following combination can make allreduce() work, but alltoall() still failed:
NCCL v2.7.8 + PyTorch v1.9.0 + CUDA 11.0
NCCL_SOCKET_IFNAME is set as “eth”, but our EFA has four eth:
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9001
        inet 10.4.2.84  netmask 255.255.224.0  broadcast 10.4.31.255
        inet6 fe80::437:f3ff:fe3a:8529  prefixlen 64  scopeid 0x20<link>
        ether 06:37:f3:3a:85:29  txqueuelen 1000  (Ethernet)
        RX packets 39803458  bytes 99075257349 (92.2 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 19278512  bytes 35978106875 (33.5 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9001
        inet 10.4.8.96  netmask 255.255.224.0  broadcast 10.4.31.255
        inet6 fe80::4a0:51ff:fecd:2c15  prefixlen 64  scopeid 0x20<link>
        ether 06:a0:51:cd:2c:15  txqueuelen 1000  (Ethernet)
        RX packets 325113  bytes 2433198989 (2.2 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 105789  bytes 7092566 (6.7 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9001
        inet 10.4.28.87  netmask 255.255.224.0  broadcast 10.4.31.255
        inet6 fe80::4de:c7ff:fe3b:1595  prefixlen 64  scopeid 0x20<link>
        ether 06:de:c7:3b:15:95  txqueuelen 1000  (Ethernet)
        RX packets 863069  bytes 6834097406 (6.3 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 298325  bytes 19788074 (18.8 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9001
        inet 10.4.18.220  netmask 255.255.224.0  broadcast 10.4.31.255
        inet6 fe80::461:3dff:fead:19bd  prefixlen 64  scopeid 0x20<link>
        ether 06:61:3d:ad:19:bd  txqueuelen 1000  (Ethernet)
        RX packets 860674  bytes 6832104026 (6.3 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 293308  bytes 19451440 (18.5 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0