Solved by numpee in post #7 
              
              
                Upon cancelling the hang, I get a timeout error: 
fd_event_list = self._selector.poll(timeout)

It seems like the processes can’t communicate with each other. So I’ve tried wrapping the training loop with model.no_sync(), and the code progresses well. 
As you mentioned, this definitely does look lik…