Your experience would mean that e.g. if you are using 10 workers, the GPU would starve after 10 batches, as each worker would have to load the next batch?
I haven’t seen this behavior yet and only see a latency in the first epoch of the DataLoader, where each worker is loading the batch. Afterwards the data loading time should be hidden in the background, if your actual training workload is large enough.