I used an IterableDataset and got
ValueError: DataLoader with IterableDataset: expected unspecified sampler option, but got sampler=<torch.utils.data.distributed.DistributedSampler object at 0x7fab6f18e898>

That workaround idea sounds good, I will try that and post back.