Solved by mrshenli in post #3 
              
              
                The new_group API requires all processes to call with the same ranks argument if even if they do not participate in the new group. See the API doc here: https://pytorch.org/docs/stable/distributed.html#torch.distributed.new_group 
In the code above, the following code breaks the above assumption. 
 …