My colleague Ross Wightman solved this through some network sleuthing. I hope this helps someone in the future!
He writes:

I spent quite a bit of time hacking around and mapping out network interfaces on the worker nodes via logging / prints in the train script.
The main network interface that is mapped to the default hostname is what we were using for the PyTorch distributed.init_process_groups TCPStore group initialization. As Cade has reported, as the nodes scaled up, quite a number of them were getting stuck trying to communicate with the master node and the whole init would fail to move forward.
I installed a PyTorch nightly env (1.11) as a PT engineer said they had some better logs in there. Slightly better I suppose, but I was just seeing socket connection timeouts being reported (which means a bunch of nodes were failing to make the connection to master). That could be many things, including routing issues, port/firewall blocking, rate limiting, capacity overload.
I started mapping out the other interfaces and figured out the pattern for the ip over infiniband interfaces. Hostname patterns as follows
enp225s0f0 - jwb0000 - the machine hostname, 2-port ethernet nic?
ib0 - jwb0000i - infiniband
ib1 - jwb0000iu - infiniband
ib2 - jwb0000iv - infiniband
ib3 - jwb0000iw - infiniband
… there are other ib interfaces as well but skipping
I assume the enp225 interface is for control / storage. I imagine it’s a 10gbe+ adapter (didn’t check yet) so should be plenty capable. Perhaps there are extra firewall rules, port range restrictions, or routing on there that impact the TCPStore handshake (any node being able to establish a TCP connection on a specified port) ?
I’m not sure if all infiniband adapters can be treated equally? I assume they are all ‘cluster interconnect’ for MPI / NCCL / etc.
I got past the startup hang issues by using the hostname for ib0 instead of the default machine hostname… basically append an ‘i’ to the hostname for MASTER_ADDR in the launch script. The nodes do start a bit staggered in that number, but everything appears to work, early nodes log that they are waiting, the laggards come online and then off to the races!
It’d be nice to have some more details on the network setup / intentions. And also what restrictions are with regards to port ranges that should be open and free to use, etc.
Also, to test this out, make sure you dial back the debugging env vars, especially TORCH_DISTRIBUTED_* vars. They force the creation of a secondary GLOO group that hangs because it tries to use the wrong interface (it goes back to the main interface because it looks up the ip itself based on socket.gethostname() – not a good idea in a large scale network).
Ross

TLDR: Ensure your $MASTER_ADDR refers to an address capable of handling all these processes, in our case that was pointing to infiniband by essentially just appending an “i” to the $MASTER_ADDR.