Hi, sorry I didn’t answer earlier, I’ll try to catch up with what was said.
As @mrshenli pointed out, the fact that RPC was attempting to use 127.0.1.1 (a localhost address) was probably due to your machine/networking not being set up in a way that allows to resolved your hostname to an IP address. Such a set-up is common in cloud providers and computing clusters but not so much in personal machines. Perhaps this should be clarified. The “workaround” is to use GLOO_SOCKET_IFNAME or TP_SOCKET_IFNAME (for ProcessGroup and TensorPipe respectively).
(Actually, right now, there’s an issue where if you use TensorPipe you need to set both those env vars)
Another thing to notice is that the IP address&port that you provide as an init_method are not in fact the ones that are used to connect the machines one to another. I realize that sounds a bit silly but the reason is one of internal code modularity: the “real” connection cannot extract such information from the init_method because that could be a file, or some other non-IP system. Hence the real connection is established in its own way, which is what lead to the localhost address being selected, and to port 2222 being “diregarded”.
The way these “real” connections are created is the same in both ProcessGroup and TensorPipe: the “server-side” of the connection opens a listening socket on an arbitrary high port, and then the client connects to that one. The port that is used is assigned automatically by the kernel among all the ones that are currently free. There is no way to predict it, limit it to a specific range, or specify it.
The “no route to host” issue seems to me to be either an issue in the routing tables (but you said you’re able to ping the machine, so that shouldn’t be the case) or a firewall prohibiting connections to the high port numbers that we use. I don’t know of a reliable way of confirming this theory other than asking your system administrator. You can however see if you can reproduce the issue using “standard” tools like netcat. In the above logs you posted, the port that was assigned by the kernel is 49552. You could set up one instance of netcat listening on that port, and another (from the other machine) connecting to it. If that work, then that’s surprising, and we should look further into what PyTorch is doing differently that prevents it from working.