Solved by LeviViana in post #2 
              
              
                I think you could use torch.distributed.launch. This snippet may be useful to you to get a better grasp on how torch.distributed.launch works.