Currently I am running on one machine, but, ideally, the solution would also be useful for both single and multiple machines.
Thanks you for the suggestions, both seem reasonable, but it might be unclear beforehand how long the processing will take (also depends on the network speed and such) so, that would require a bit of tweaking for the timeout parameter. I also do not known beforehand how large the dataset will be precisely, as it can happen new samples have been added so it would be tricky to write a class which effectively splits the dataset into smaller ones, as that would require me to know how large it.
So, perhaps, a combination of 1 and 2 would also work: I can make a rough split across WORLD_SIZE, and base this on get_rank. So, it can happen that some processes are finished earlier than the others. If I then call torch.distributed.barrier() at the end of the processing of the  dataset, this would have the effect that the preprocessing will be split  along the processes and all will wait until each one is done with their part. Do I understand this correctly?