On the topic of time savings with data-parallel, it’s entirely plausible that the real bottleneck with your setup is data augmentation and preparation which takes place on the CPU. I’ve found that ToTensor() on the cpu takes about a second per 1k images. Your epoch takes 8 seconds for 6k.