ptrblck:

( num_workers>0 in your DataLoader ) you can load and process your data while your GPU is still busy training your model, thus possibly hiding the loading and processing time of your data


Same behaviour even with num_workers=0 or 1



 ptrblck:

Since you apply Normalize(mean=(0.5, 0.5, 0.5), std=(0.5, 0.5, 0.5)) afterwards, you’ll get the new range [-1, 1]


Oops, good catch, just missed that!



 ptrblck:

I guess by “full batch”, you’ve specified the batch_size=60000 ?


Yup!



 ptrblck:

Did you push all your data samples to the GPU beforehand?


That happens inside the epochs loop. Is that bad?
Besides I don’t see any of the above increasing so much the time of DataLoader. I would have expected using DataLoader and batch size 60000 to be equivalent in terms of speed with my own testing when I’m not using the DataLoader but the full dataset?  It’s hard that this would happen by only by the answer you mentioned on point 1).