Hello, I am trying to train ImageNet on a 8GPU machine with DDP mode. However, my machine is not good at reading large scale small files. I have to make a tar file of the whole dataset (130Gb), read the tar file into memory and extract them in memory. I have a CPU memory of 360Gb. So it would be OK to use DataParallel mode. But it seems I cannot use DistributedDataParallel since I need to load the dataset 8 times. Is there any method that I can train the model with DDP mode?
Thanks!