I have a similar question here. If this is the way the DistributedSampler sample data, and if the portion of the data is quite smal on each single GPU, the network on each single GPU may overfit trained on the small portion of the dataset.
Will the portion of the dataset on each single GPU update after each epoch?