Solved by mrshenli in post #2 
              
              
                IIUC, this is typical DistributedDataParallel training? If so, yes, PyTorch natively support that. Here is another tutorial.