No I haven’t, I will try that next thanks!
That said I have already tested preloading entire dataset to GPUs. So aside from data loading, I know PyTorch recommends using DataParallelDistributed, but what else could there be that is causing such small speed gains from x4 parallelism?