It seems to do the right thing using the barrier code - same as yours (I added some debug prints to it):
        import torch
        if training_args.local_rank > 0:
            logger.info('Waiting for main process to perform the mapping')
            torch.distributed.barrier()

        print(f"START PREPROCESSING on RANK {training_args.local_rank}")
        train_dataset = train_dataset.map(
            preprocess_function,
            batched=True,
            num_proc=data_args.preprocessing_num_workers,
            remove_columns=column_names,
            load_from_cache_file=not data_args.overwrite_cache,
            desc="Running tokenizer on train dataset",
        )
        print(f"END PREPROCESSING on RANK {training_args.local_rank}")

        if training_args.local_rank == 0:
            logger.info('Loading results from the main process')
            torch.distributed.barrier()

Here is the log with some new lines added to make it easier to read:
START PREPROCESSING on RANK 0
Running tokenizer on train dataset:   0%|                                                                                                  | 0/1 [00:00<?, ?ba/s]
06/24/2021 19:21:09 - INFO - datasets.arrow_dataset -   Caching processed dataset at /home/stas/.cache/huggingface/datasets/wmt16/ro-en/1.0.0/0d9fb3e814712c78517
6ad8cdb9f465fbe6479000ee6546725db30ad8a8b5f8a/cache-1bb178ab5058b59a.arrow
Running tokenizer on train dataset: 100%|██████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 48.21ba/s]
06/24/2021 19:21:09 - INFO - datasets.arrow_writer -   Done writing 500 examples in 113764 bytes /home/stas/.cache/huggingface/datasets/wmt16/ro-en/1.0.0/0d9fb3e
814712c785176ad8cdb9f465fbe6479000ee6546725db30ad8a8b5f8a/tmp34tbvrkc.
END PREPROCESSING on RANK 0

06/24/2021 19:21:09 - INFO - __main__ -   Loading results from the main process


START PREPROCESSING on RANK 1
06/24/2021 19:21:12 - WARNING - datasets.arrow_dataset -   Loading cached processed dataset at /home/stas/.cache/huggingface/datasets/wmt16/ro-en/1.0.0/0d9fb3e81
4712c785176ad8cdb9f465fbe6479000ee6546725db30ad8a8b5f8a/cache-1bb178ab5058b59a.arrow
END PREPROCESSING on RANK 1

this is on 2 gpus torch distributed.
So the logs look correct to me. The 2nd process loads the cache.
note, I deleted the cache before running the script, i.e.:
rm /home/stas/.cache/huggingface/datasets/wmt16/ro-en/1.0.0/0d9fb3e814712c785176ad8cdb9f465fbe6479000ee6546725db30ad8a8b5f8a/cache-1bb178ab5058b59a.arrow

so that the log shows that the cache was created by the first process. otherwise it too will load the cache.
We are just going to make a context manager to make it easier.
Also make sure you’re on the latest datasets.