I met the same problem with only one GPU(NVIDIA GTX 1080 Ti) while training  on ImageNet dataset with ImageFolder API.
In my case, the dataset images are on my Seagate 2TB HDD so the problem may partly caused by the slow reading speed of HDD. Probably putting the dataset on a SSD may help a lot.
However I found it got better when setting shuffle=False and num_workers larger but the problem still happened sometime. I think the shuffle procedure in Torch may also be the bottleneck.