I edited my question by showing how I measure the loading time (see the above pseudocode). I also changed the number of workers (2, 3, 4, 6, 12), in each case the first dataloader is always the slowest one to return batches. However, this is not the case when num_workers=1, in which case the first dataloader is not always the slowest. For example, with num_workers = 1 I get these results:
#Epoch 0
Avg time batch1:  1455.2
Avg time batch2:  274.23

#Epoch 1
Avg time [ms] batch1:  35.30
Avg time [ms] batch2:  1379.09

#Epoch 2
Avg time [ms] batch1:  973.86
Avg time [ms] batch2:  716.08

#Epoch 3
Avg time [ms] batch1:  8.59
Avg time [ms] batch2:  1817.66

#Epoch 4
Avg time [ms] batch1:  1077.48
Avg time [ms] batch2:  787.06

#Epoch 5
Avg time [ms] batch1:  25.82
Avg time [ms] batch2:  1840.94

Thus, with num_workers = 1 one of the two dataloaders is often much slower than the other, though it’s not always the first dataloader to be the slowest. I should also specify that I am using a google compute engine (i.e. a google cloud machine with just 2 CPUs and a Tesla K80 GPU). I don’t understand why there is such a big difference in terms of loading time between the two dataloaders. Ideally, I would like them to take the same amount of time (and possibly the shortest time). What should I do in this sense? If I merge both datasets into one unique dataloader, would that make any difference?