The periodical slowdown every num_workers iterations points towards the aforementioned loading of a full batch in each worker.
In your current script you don’t have any workload, so it’s expected that you would see slower iterations once the queue is empty.
The timings would refer to:
10.58901309967041 # no batches ready in the queue, all 4 workers are preloading
1.8810162544250488 # Grab next batch from queue. Since this is slower than the next step the queue might not have been filled yet.
0.009713411331176758 # grab next batch from queue
0.7070198059082031 # grab next batch from queue
10.15380048751831 # Queue is empty, as no worker was able to load a full batch yet

If your model training is sufficiently large, the DataLoader might be able to preload the next batch(es) while your model is training.