Thanks for sharing the faulthandler output, that does definitely help in narrowing down where the problem might come from. However, the place where the nodes are stuck is where rest of the logic resides in C++ and hence the python stack doesn’t cover this. In particular, the following would help in debugging this further:

If possible, get access only to Rank 0 of the compute node and get a full gdb stack trace as I mentioned above.
When we print Starting rank, let’s also print args.dist_backend, args.dist_url and args.world_size there as well for that particular rank. In particular, I’m interested in rank 0 here as well.
Set torch._C._set_print_stack_traces_on_fatal_signal(True) for rank 0 and see if it produces C++ stack traces? If not, we probably need to kill rank 0 with a fatal signal (ex: kill -9) to produce a C++ stack trace for that particular rank.

Another thing, you could try if possible is to manually check TCP connectivity between rank 0 machine and another machine whose rank is stuck in the TCP rendezvous. For example, check if the stuck rank can reach MASTER_ADDR:MASTER_PORT on rank 0.