torch.utils.data.distributed.DistributedSampler uses the epoch number as a seed to deterministically randomize the data to create coherent bigger batches (i.e. a different subsample of the dataset is seen each time through training and the samples do not overlap between the different training instances.)
Also do you scale the learning rate according to the number of instances you spawn ?