Thanks for your response. I use a machine with 48 threads and 256G RAM so I preload the data and set worker=48.
I try your code and get similar result. It seems that worker=0 does not feed gpu enough, lower than 30%. If I use 48 workers, the feed speed is very fast, but the initialization of the iterator is slow. For example, if I use
for i, d in enumerate(dataloader_train):
print(i, d.shape)
on 48 workers, it will take about 30s before print anyting. After that it is very fast since 48 workers start to work. If I use 0 workers, then there is almost no initialization time but print part is slow.
I guess the multiprocessing under the dataloader is not suitable on preloaded very large dataset?