Can you try specifying GLOO_SOCKET_IFNAME to select the appropriate interface on each node as described here: https://pytorch.org/docs/stable/distributed.html#choosing-the-network-interface-to-use 51?