@ohm The AWS tutorial goes over how to use the DistributedSampler to split your dataset into even parts: https://pytorch.org/tutorials/beginner/aws_distributed_training_tutorial.html#initialize-dataloaders 5. This is not an official PyTorch tutorial, but the “With multiprocessing” section in https://yangkky.github.io/2019/07/08/distributed-pytorch-tutorial.html 1 describes how to use DistributedSampler to do this for MNIST.