As far as I understand, you need to give the reduced batch size to DataLoader. For examples, if total batch size is 256 and 4 gpus, then the given batch size is 64. Here is the example of code: link 3
In DIstributedDataSample, each process has different rank, so sampled inputs will be different. In detail, the initial point of sampling is shifted according to rank, so subsampled indices won’t be overlapped. (here is the line of the code 3)