Assuming you are using a DistributedSampler in your DistributedDataParallel use case you could try to cache the sbsets only in each process.
However, note that ImageNet is a large dataset so you would need to use a lot of host memory to store it. I’m not familiar with your approach, but in case you want to create caches and directly reuse them, note that data augmentation would most likely be disabled (unless you want to transform the samples again), which might hurt your training performance.