Solved by andfoy in post #3 
              
              
                @dashesy You can use torch.utils.data.distributed.DistributedSampler (docs) to prevent input data broadcasting among nodes. You can see it in action in the distributed ImageNet example