For a similar issue, using args.workers=8 worked for me. My data load times went down from 3 seconds to ~0 seconds. I’m using a modified version of the pytorch imagenet example, with 3 GPUs on the same server. The num_workers passed to the DataLoader is computed as int((args.workers + gpus_per_node - 1)/gpus_per_node), which is int((8 + 3 - 1)/3), or 3 DataLoader workers per GPU.