I have a computation bound augment process, and I set num_workers=50.
Then there’s a obvious delay in data loading every 50 batches. It seems that each worker prepare only  one batch and wait still until next invocation. CPU and IO utilities are all low.
I use the imagenet example.
Here is some logging.
Epoch: [0][0/1124]      Time 100.303 (100.303)  Data 95.814 (95.814)    Loss 2.1197 (2.1197)    Acc@1 1.562 (1.562)     Acc@2 51.562 (51.562)
Epoch: [0][10/1124]     Time 0.422 (9.584)      Data 0.011 (8.753)      Loss 2.2578 (2.1322)    Acc@1 4.297 (4.865)     Acc@2 6.250 (20.312)
Epoch: [0][20/1124]     Time 0.288 (5.191)      Data 0.078 (4.606)      Loss 2.3653 (2.1892)    Acc@1 16.406 (7.310)    Acc@2 63.281 (21.001)
Epoch: [0][30/1124]     Time 0.288 (3.604)      Data 0.071 (3.134)      Loss 2.2124 (2.1810)    Acc@1 23.047 (10.144)   Acc@2 46.094 (28.679)
Epoch: [0][40/1124]     Time 0.295 (2.795)      Data 0.094 (2.382)      Loss 2.2084 (2.1984)    Acc@1 2.344 (11.719)    Acc@2 3.906 (30.164)
Epoch: [0][50/1124]     Time 0.434 (2.312)      Data 0.098 (1.927)      Loss 2.0883 (2.1817)    Acc@1 27.344 (11.060)   Acc@2 50.391 (28.592)
Epoch: [0][60/1124]     Time 30.907 (2.497)     Data 30.285 (2.119)     Loss 2.0824 (2.1731)    Acc@1 19.531 (12.897)   Acc@2 41.406 (31.276)
Epoch: [0][70/1124]     Time 0.816 (2.326)      Data 0.102 (1.933)      Loss 2.0904 (2.1663)    Acc@1 22.656 (14.277)   Acc@2 43.359 (33.291)
Epoch: [0][80/1124]     Time 0.192 (2.094)      Data 0.065 (1.707)      Loss 2.1167 (2.1607)    Acc@1 21.094 (15.355)   Acc@2 42.188 (34.891)
Epoch: [0][90/1124]     Time 0.788 (1.904)      Data 0.168 (1.528)      Loss 2.2529 (2.1558)    Acc@1 25.781 (16.303)   Acc@2 46.484 (36.178)
Epoch: [0][100/1124]    Time 0.324 (1.749)      Data 0.212 (1.385)      Loss 2.1106 (2.1537)    Acc@1 21.484 (16.901)   Acc@2 48.438 (37.233)
Epoch: [0][110/1124]    Time 0.414 (1.633)      Data 0.002 (1.267)      Loss 2.0465 (2.1491)    Acc@1 23.438 (17.325)   Acc@2 48.438 (37.968)
Epoch: [0][120/1124]    Time 45.406 (1.906)     Data 44.589 (1.537)     Loss 2.2800 (2.1496)    Acc@1 20.703 (17.859)   Acc@2 42.578 (38.598)
Epoch: [0][130/1124]    Time 0.591 (1.824)      Data 0.007 (1.454)      Loss 2.0338 (2.1466)    Acc@1 19.141 (18.079)   Acc@2 45.703 (39.054)
Epoch: [0][140/1124]    Time 0.510 (1.765)      Data 0.184 (1.397)      Loss 2.1249 (2.1457)    Acc@1 21.875 (18.426)   Acc@2 49.609 (39.583)
Epoch: [0][150/1124]    Time 0.203 (1.670)      Data 0.004 (1.308)      Loss 2.1863 (2.1450)    Acc@1 20.703 (18.755)   Acc@2 42.188 (39.929)
Epoch: [0][160/1124]    Time 0.269 (1.589)      Data 0.084 (1.231)      Loss 2.2051 (2.1434)    Acc@1 23.828 (19.031)   Acc@2 48.047 (40.302)
Epoch: [0][170/1124]    Time 0.281 (1.520)      Data 0.077 (1.163)      Loss 2.1192 (2.1403)    Acc@1 21.875 (19.301)   Acc@2 44.531 (40.634)
Epoch: [0][180/1124]    Time 40.498 (1.675)     Data 39.783 (1.321)     Loss 2.0573 (2.1410)    Acc@1 24.609 (19.572)   Acc@2 47.656 (40.981)