Good question!
From the docs:

In this mode, each time an iterator of a  DataLoader  is created (e.g., when you call  enumerate(dataloader) ),  num_workers  worker processes are created. At this point, the  dataset ,  collate_fn , and  worker_init_fn  are passed to each worker, where they are used to initialize, and fetch data. This means that dataset access together with its internal IO, transforms (including  collate_fn ) runs independently in the worker process.

This implies:


when num_workers=0, for each epoch, data fetching is done by MAIN process


when num_workers=1, for each epoch, MAIN spawns a new worker each time enumerate(loader) is called. At each epoch, you should expect output of 7,8,9


when num_workers=2, for each epoch, MAIN spawns 2 new workers each time enumerate(loader) is called. Since len(dataset)=3, worker1 gets 2 indices, worker2 gets 1 index. You should expect output of 7,8 from worker1, and 7 from worker2.


If you change len(dataset) to 4,

with num_workers=2, you should expect 7,8,7,8
with num_workers=3, you should expect 7,8,7,7



To bypass this GIL limitation, you can look into multiprocessing communication constructs.