Solved by mrshenli in post #4 
              
              
                If you have 8 processes (4 processes per node with 2 node), world_size should be 8 for init_process_group. But you don’t need to set that, as the launcher script will set the env vars for you properly. 

I am not sure what I should set as  RANK  but I set it as 0 and 1 for the two nodes 

There are…