I am really not an expert on how to use DDP but from what I understand, each process has its own dataset/dataloder and they use a distributed sampler to make sure they don’t sample the same thing: torch.utils.data — PyTorch 1.7.1 documentation 2