@rwightman, @songyuc I did some experimentation with number of workers, and I can say that the best way to find the optimal one is to run a test over a range of values, for maybe 100 batches. It is highly dependent on particular combination of a CPU, storage, dataloader type, preprocessing methods, and model type/size. I’m afraid there’s not magic formula to rely on. For example, I found that on my 6 core CPU, with DataParallel, standard Imagenet preprocessing for Resnets, vanilla Resnet-18 model, and Samsung 850 Pro SSD, DALI-CPU works best, and the optimal number of workers is 12. But the range 8-16 is nearly as fast, which is counterintuitive given number of cores/threads.
P.S. and the difference between optimal and default params can be quite significant - for example, in the above scenario, using the default num_workers=4 will lead to ~20% slow down.