Hi,
I am currently playing around distributed training, and I have some trouble trying to understand it:
When using torch.utils.data.distributed.DistributedSampler, does it mean to split the whole dataset D to D1 and D2(considering the one node two gpus scenario), and each process only iterates data from one of D1 or D2, thus there will never be a batch generated from a mixed of D1 data and D2 data? Or will the split operation be committed automatically each time one epoch is done and another epoch is started?