If you are using nn.DistributedDataParallel, each process could only load the  (subset of the) data. nn.DataParallel creates model replica on each device for each forward pass, splits the data tensor in the batch dimension (dim0) and sends a chunk of the data to each device.
I’m not sure which approach you are using, but DDP should be faster.