kirk86:

It doesn’t seem like the case here. I mean comparing DataLoader using full batch with simply full batch in my case without the DataLoader, in both cases same transformations. Still full batch without DataLoader is almost half the time that it takes the DataLoader with full batch, and here is where I don’t understand why so much discrepancy?


The speed difference is because you are loading the whole dataset from disk. Also, naturally, the larger the dataset the longer the loading time. In a real-world scenario more than one batch will usually loaded before the network finishes the the forward and backward loop. To make your benchmark fair,
a) use a dataloader with a reasonable batchsize and reasonable number of workers and train the network
b) compare it with a network where your dataset is in memory like right now, but iterate over it with the same batch size.



 kirk86:


images = images.view(images.size(0), -1)
temp = torch.cat([ images, labels.view(images.size(0), 1).to(torch.float) ], dim=1)
idx = torch.randperm(temp.nelement()) temp.view(-1)[idx].view(temp.size())
images = temp[:, :images.size(1)] labels = temp[:, -1].to(torch.long)

I dunno if there’s a more efficient way?


Cat is naturally very expensive. Just keep an index over all your images that you shuffle and then use it as a mask instead of making new arrays each time. The number of images doesn’t change, so you can simply reuse (reshuffle) that index instead of creating new arrays each time.
PS: I see that you are using 10 workers. That is probably too much and will slow you down due to the communication overhead between processes. I think 4 is a reasonable number because the dataset is rel small.