I think the workers are not fast enough to provide the next batches, so that you have to wait for the data loading. I don’t think this issue is due to respawning the workers, but rather it seems you have an IO bottleneck.
You could try to play around with the number of workers or to speed up the data loading (e.g. move the data to an SSD if it’s stored on an HDD).