It looks like the workers are not ready loading the next batches.
Are you using a HDD or SSD in your system?
Also, are you just calling the DataLoader or do you also have a workload (training) in the loop?
Usually you can mask some of the data loading time with the training procedure.