nn.DataParallel will clone the model onto al specified devices, split the input batch in dim0, and send each chunk to the corresponding device. It won’t do anything with your data loading. A good description is given in this blogpost. If you want to use load a subset of the data for each device, use DistributedDataParallel with the DistributedSampler, which would also yield a larger speedup.