Upon cancelling the hang, I get a timeout error:
fd_event_list = self._selector.poll(timeout)

It seems like the processes can’t communicate with each other. So I’ve tried wrapping the training loop with model.no_sync(), and the code progresses well.
As you mentioned, this definitely does look like a setup issue.
Any ideas what might be causing communication issues locally?
By the way, thanks for you help 
Edit I have found that using the gloo backend is a temporary fix to this issue. However, there is no noticeable speedup when using single GPU vs 2 GPUs w/ Gloo. Usually there is quite a noticeable speedup.
Edit 2 Disabling P2P with export NCCL_P2P_DISABLE=1 seems to solve the issue as well, using direct shared memory.