Fast data loader for Imagenet
longcw
How many workers are you using? What kind of disk are you using to store the dataset?
apaszke
longcw
Yes, the data transforms can be quite expensive.
apaszke
Unfortunately all those mass consumption dataset loaders and feeders are far from optimal, everyone serious about this ends up writing their own performant code (or compensate by getting expensive SSDs and CPUs) but don’t publish it because it’s embarrassingly written. People often say you need to use SSDs and whatnot, but I use a couple of ancient HDDs and it’s enough to feed a duo 1080 setup. It’s just troublesome to write a properly multi threaded application for this, you don’t even need to do it in C if you have a good CPU.
Veril
More workers seems slower. The data loading is extremely slow with a low cpu load, which means the dataloader can’t make full use of the cpu.
Chong_Lv
gngdb
did you store the images file after preprocessing?
lkywk
Yeah, but it’s a one-off cost. I just copied it to the other servers after.
gngdb
I run the script, but I meet the problem.
lkywk
Could be that you’re missing protobuf. You could install it with conda. If that works, I’ll add it to the README, I don’t have time to test it myself right now.
gngdb
I fixed it days ago. I think it is caused by protobuf.
lkywk
Hi, I’ve used your data but the loading speed is still 4s for batch=256, num_worker=1, which I think is not fast enough, is it normal?
charlesjiangxm
I used 4 workers, and each minibatch took 0.59s to process, including the time for forward and backward propagation. Maybe you should try with more workers?
gngdb
Hi, I use your method. But when I train the imagenet, I meet the problem
Chenzhi_Jiang
I fixed it. Just need to downgrade msgpack to 0.5.6
Chenzhi_Jiang