Solved by teng-li in post #7 
              
              
                @leo-mao, you should not set world_size and rank in torch.distributed.init_process_group, they are automatically set by torch.distributed.launch. 
So please change that to dist.init_process_group(backend=backend, init_method=“env://”) 
Also, you should not set WORLD_SIZE, RANK env variables in your …