I had a similar use case and ended up implementing an IterableDataset that handles both multiprocessing via DataLoader and distributed training.
I shared my code here 245.
But I have to say that I turned away from that approach in the end, as it is relatively inflexible and turned the IterableDataset simply into an indexable one.