In my case, the ImageNet dataset is on my local HDD. I use torchvision.datasets.ImageFolder and torch.utils.data.DataLoader  with batch_size=32 shuffle=True num_workers=16 to load images. Only randomcrop is applied on the images for preprocessing.
The extra time T it costs for every Nth iteration is directly proportional to the thread number N.
T is about 15s or more when batch_size=32 and   num_workers=16.