If the workers aren’t able to pre-load the batches in time, you would see spikes once in a while (or every num_workers iterations), but your issue sounds a bit strange, since the data loading speed tanks towards the end. Are you seeing the same slowdown in each epoch?