mrshenli:

init_process_group


Thank you very much for your reply.
I checked by printing the NCCL_SOCKET_IFNAME in my python script like this:
        os.environ.setdefault("NCCL_DEBUG", "INFO")
        os.environ.setdefault("NCCL_IB_DISABLE", "1")
        print("Setting NCCL_SOCKET_IFNAME")
        os.environ.setdefault("NCCL_SOCKET_IFNAME", "^lo,docker")
        print("Finished NCCL_SOCKET_IFNAME")
        print(os.environ.get("NCCL_SOCKET_IFNAME"))

        # See:
        # https://pytorch.org/docs/stable/distributed.html#torch.distributed.init_process_group
        os.environ.setdefault("NCCL_BLOCKING_WAIT", "1")
        print("self.dist_backend:",self.dist_backend)
        print("self.dist_init_method:", self.dist_init_method)
        print("self.dist_world_size:",self.dist_world_size)
        print("self.dist_rank:",self.dist_rank)
        torch.distributed.init_process_group(
            backend=self.dist_backend,
            init_method=self.dist_init_method,
            world_size=self.dist_world_size,
            rank=self.dist_rank,
        )


the output of one node is like this:
Setting NCCL_SOCKET_IFNAME
Finished NCCL_SOCKET_IFNAME
^lo,docker
self.dist_backend: nccl
self.dist_init_method: file:///home/storage15/huangying/tools/espnet/egs2/voxforge/asr1/vox.init
self.dist_world_size: 3
self.dist_rank: 2
MASTER_ADDR : None
MASTER_PORT : None
Besides , I use ifconfig  to check the network  of  one of the failed node:
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
inet 10.38.10.174  netmask 255.255.255.0  broadcast 10.38.10.255
inet6 fe80::9a03:9bff:fe0e:ff72  prefixlen 64  scopeid 0x20
ether 98:03:9b:0e:ff:72  txqueuelen 1000  (Ethernet)
RX packets 265553414516  bytes 374221992683088 (340.3 TiB)
RX errors 0  dropped 827204  overruns 0  frame 0
TX packets 101116217165  bytes 122158006104703 (111.1 TiB)
TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
eth1: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
ether 98:03:9b:0e:ff:73  txqueuelen 1000  (Ethernet)
RX packets 0  bytes 0 (0.0 B)
RX errors 0  dropped 0  overruns 0  frame 0
TX packets 0  bytes 0 (0.0 B)
TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
inet 127.0.0.1  netmask 255.0.0.0
inet6 ::1  prefixlen 128  scopeid 0x10
loop  txqueuelen 0  (Local Loopback)
RX packets 17752724  bytes 17788869931 (16.5 GiB)
RX errors 0  dropped 0  overruns 0  frame 0
TX packets 17752724  bytes 17788869931 (16.5 GiB)
TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0