Thank you @iffiX for the insightful response. I am not sure if I fully understood, but I do have:
    if local_rank != 0:
        torch.distributed.barrier()

earlier in the code. The purpose is to pause the execution of all the local ranks except for the first local rank to create directory and download dataset without conflicts. Once the first local rank completed the download and directory creation, the reset of local ranks could use the downloaded dataset and directory.
In your opinion, how should I modify my code in particular? Thank you.
Best,
Lei