Hi
I want to know how to speed up the dataloader. I am using torch.utils.data.DataLoader(8 workers) to train resnet18 on my own dataset. My environment is Ubuntu 16.04, 3 * Titan Xp, SSD 1T.




Epoch: [1079][0/232]
Time 5.149 (5.149)
Data 5.056 (5.056)
Loss 0.0648 (0.0648)
Prec@1 98.047 (98.047)




Epoch: [1079][10/232]
Time 0.208 (0.778)
Data 0.001 (0.610)
Loss 0.0904 (0.0833)
Prec@1 95.312 (96.165)


Epoch: [1079][20/232]
Time 0.206 (0.599)
Data 0.001 (0.421)
Loss 0.0771 (0.0877)
Prec@1 96.484 (95.964)


Epoch: [1079][30/232]
Time 0.208 (0.529)
Data 0.074 (0.353)
Loss 0.0681 (0.0886)
Prec@1 98.047 (96.006)


Epoch: [1079][40/232]
Time 1.799 (0.538)
Data 1.697 (0.376)
Loss 0.0647 (0.0860)
Prec@1 96.484 (96.103)


Epoch: [1079][50/232]
Time 0.201 (0.516)
Data 0.001 (0.360)
Loss 0.0578 (0.0836)
Prec@1 96.484 (96.186)


Epoch: [1079][60/232]
Time 0.443 (0.494)
Data 0.328 (0.340)
Loss 0.0602 (0.0835)
Prec@1 97.656 (96.203)


Epoch: [1079][70/232]
Time 0.201 (0.483)
Data 0.001 (0.334)
Loss 0.1140 (0.0858)
Prec@1 94.922 (96.121)


Epoch: [1079][80/232]
Time 0.616 (0.489)
Data 0.521 (0.343)
Loss 0.0932 (0.0857)
Prec@1 95.703 (96.161)


Epoch: [1079][90/232]
Time 0.200 (0.485)
Data 0.001 (0.341)
Loss 0.0596 (0.0847)
Prec@1 98.047 (96.175)


Epoch: [1079][100/232]
Time 0.362 (0.477)
Data 0.269 (0.334)
Loss 0.0877 (0.0854)
Prec@1 94.531 (96.117)



The log shows that the dataloader takes at least 50% time of the training process.  So I want to speed up the training process by reducing the time for dataloader.
I analyses the time for the datalayer get_item()
total time: 0.02
load img time: 0.0140, 78.17%
random crop and resize time: 0.0001, 0.68%
random flip time: 0.0001, 0.40%
other time:  22.36%
It shows that the reading image takes the most of time (I read 3 images in this section). I uses the codes from torch.utils.data.Dataset
def pil_loader(path):
# open path as file to avoid ResourceWarning (https://github.com/python-pillow/Pillow/issues/835 40)
with open(path, ‘rb’) as f:
with Image.open(f) as img:
return img.convert(‘RGB’)
Therefore, I tried Rookie ask: how to speed up the loading speed in pytorch 593 . I save the images as strings with pickle in lmdb. Then I load them back. I found that it doesn’t speed up too much. Maybe pickle.loads() still costs too much time.
Now I have no idea to speed up the dataloader.  Any hints will help me much.
Thanks.