I don’t know if you solved this problem, but I have noticed this happening to me as well. I could actually tell something was wrong just by comparing the time between various length datasets, reducing the size of the same dataset, and reducing/increasing the number of workers.
Basically, using COCO dataset wrapped into a torch.DataSet, I noticed that by changing the total number of images the time was not reduced; also changing the batch size was not affecting the initial batch loading time, and, definitely, increasing the number of workers was dramatically raising this loading time.
Finally, I understood that the cause of this was the way of spawning the processes at the beginning, which I replicated by looking at the sample code of PyTorch Imagenet. In this, “torch.multiprocessing.Spawn” is used, but it should be avoided. The best practice, and might be unique to date, is to rely on the official launch module, which relies on subprocess.Popen function. I do not know the precise difference and this should be further investigated.
Hope this helps someone else spare some time either in training or in looking for a solution.