Hi @ptrblck. Thanks for you feedback.
I was expecting the same, and to test that, I ran the sequential model with 10 data loading workers. It uses the same dataloader in the same way. But there, I did not see  a similar behavior.




batch
time on next(…)




0
5.76223


1
0.71920


2
0.00025


3
0.00025


…
…


10
0.00083


11
0.00077


12
0.00030


13
0.00027


…
…


20
0.00056


21
0.00029


22
0.00032


23
0.00031


…
…


30
0.00035


31
0.00030



Coming back to your point, shouldn’t the data loading threads load batches ahead of time? Is this an expected behavior for the multi-process dataloader (slow-down in every num_workers batch)?
PS: Clarification on the usage here. In Pipedream, rank 0 worker would be the only one loading data from the dataset, and theoretically, when it comes to data loading, there’s no difference between rank0 worker of pipedream and sequential execution. Main difference would be pipedream rank0 worker would be doing 1/world_size amount work.