Thanks @ptrblck, so if the training will be different processes than multiprocessing dataloader, right? from your description, if training is slower than dataloading, then basically we should get continuous training and loading time will he shaded?
Also if I use Data parallel, and based on understanding data parallel is using multi threading, so how this multi threading data parallel will work with multi process data loader? still the same way multi process data loader loads the data into queue, and training process(a different process) spin multi threads according to multi GPU to train ?