I checked for batch size = 4. If I run such a code:
batch_size = 4
trainloader = torch.utils.data.DataLoader(dataset, batch_size=batch_size, num_workers=4)
dataiter = iter(trainloader)
for i in range(8):
    start = time.time()
    q, p, n = dataiter.next()
    q, p, n  = q.to(device), p.to(device), n.to(device)
    end = time.time()
    print(end - start)

I will get such time results:
10.58901309967041
1.8810162544250488
0.009713411331176758
0.7070198059082031
10.15380048751831
2.085867404937744
0.009247779846191406
0.009662628173828125

I mean this is ok compared to one worker, but still, I don’t know what is happening inside the loader so that it needs 10 seconds instead of 0.3. Maybe it is because workers collecting all samples as you explained, but for one worker problem stays