Hey @nash, thanks for sharing the code. The code looks correct to me, except the rank argument in DistributedSampler might need to be global rank (i.e., current_env["RANK"]) instead of the local rank? But this is not the cause of the error, as the error was thrown in DDP ctor when it tries to run broadcast.
Regarding the error:

Curious, does gloo backend work?
Could you please print out the env vars set by the launching script? Sth as what this example 4 does to env_dict.
Could you try run the following command on both nodes? If the return IP is not what you intended, you might need to set either GLOO_SOCKET_IFNAME or NCCL_SOCKET_IFNAME as mentioned here 9 depending on which backend you are using.getent hosts `hostname`