I have two datasets, each one having 60k RGB images (320x320) saved as .pt files, and two DataLoaders (one for  each dataset). Both DataLoaders have equal parameters: batch size=256, num_workers = 6, shuffle = True, drop_last=True. At each epoch, I get batches from the DataLoaders as in this pseudocode:
data_generator1 = data.DataLoader(...)
data_generator2 = data.DataLoader(...)

num_batches = len(data_generator1) # note: generators have same len

# tools for measuring execution times
start_batch1 = torch.cuda.Event(enable_timing=True)
end_batch1 = torch.cuda.Event(enable_timing=True)
start_batch2 = torch.cuda.Event(enable_timing=True)
end_batch2 = torch.cuda.Event(enable_timing=True)

for epoch in range(epochs):
    t_batch1 = []
    t_batch2 = []
    generator_iterator1 = iter(data_generator1)
    generator_iterator2 = iter(data_generator2)
    for i in range(num_batches):
        try:
            # Get a batch from both dataloaders
            start_batch1.record()
            batch1, labels1 = next(generator_iterator1)
            end_batch1.record()
            torch.cuda.synchronize()
            t_batch1.append(start_batch1.elapsed_time(end_batch1))

            start_batch2.record()
            batch2, labels2 = next(generator_iterator2)
            end_batch2.record()
            torch.cuda.synchronize()
            t_batch2.append(start_batch2.elapsed_time(end_batch2))
        except StopIteration:
             # dataloaders are empty
             break

        # Train model....

    print("Avg time batch1: ", sum(t_batch1)/len(t_batch1))
    print("Avg time batch2: ", sum(t_batch2)/len(t_batch2))

Now, the problem is that batch1, labels1 = next(generator_iterator1) on average is 5x slower than batch2, labels2 = next(generator_iterator2), i.e. 1500 millseconds vs 300 milliseconds (with 60000/256 = 234 batches in total, this results in 5.85 minutes vs 1.17 minutes per epoch). Of course, I would like both to take the minimum amount of time in order to speed up the training process. At first I thought the problem was on the data (maybe dataset1 is “heavier” than dataset2, or maybe I saved them in the .pt files as different types of data). However, I swapped the commands order in the code by placing batch2, labels2 = next(generator_iterator2) first and then batch1, labels1 = next(generator_iterator1). And guess what? Now batch2, labels2 = next(generator_iterator2) is 5x slower. So it’s clearly not the kind of data, but rather which DataLoader is fetched first. In other words, no matter which dataloader I fetch, the first dataloader to be fetched is always the slowest to return the batches. Does anybody know why this is happening?
EDIT: I forgot to say that I am using pytorch on Ubuntu.