You could do one of two things:

Segment your input dataset into WORLD_SIZE chunks and let every process preprocess its own subset of the dataset (this also gets you parallelization of the preprocessing). You can call torch.distributed.get_rank 2 to get the rank of the current process and use this to index into your input dataset.
Like you say, force rank == 0 to perform all preprocessing, and make all other workers wait for completion. You can do this with a call to torch.distributed.barrier 100. The default timeout for this call is 30 minutes. If preprocessing takes more time, you can tune the timeout through the timeout kwarg to torch.distributed.init_process_group 19.

Are you running on a single machine or multiple machines?