Here is the overview for the distributed training tools offered by PyTorch: https://pytorch.org/tutorials/beginner/dist_overview.html 2
If you are looking for data parallel training, you might want to start from DataParallel?