I am training a resnet-18 network in pytorch with a standard pytorch dataloader. When I increase the number of workers from 0 (no parallelization) to any number (even just to 1), the time for loading a batch increases by a large factor (~x5), instead of decreasing. My CPU utilization is low, and the images are loaded from my local SSD.
Any idea how to overcome this issue?