This is good place to start:

  
      
      github.com
  
  
    

pytorch/examples 39

master/imagenet

  A set of examples around pytorch in Vision, Text, Reinforcement Learning, etc.

  
  
    
    
  
  


The example script and README show how to setup multi-node training for ImageNet. You may also want to try out PyTorch Lightning which has a simple API for multi-node training:
https://pytorch-lightning.readthedocs.io/en/stable/multi_gpu.html 16