I am also suffering this issue. When I set num_workers to 0, this issue goes away. But with larger num_workers, the time between epochs is longer, especially when batchsize is larger like 256.
Did dataloader repawn processes everytime it runs out of data like when epoch ends?
Its spawning overhead takes more time than the train itself. I think it should be optimized.