In each epoch the workers will recreate a batch, so the first iteration might be slower.
Especially if you are using a lot of workers with a large batch size.
IIRC there is an open PR to use multiple workers to create a single batch, which would speed up the first iteration.