Solved by mrshenli in post #4 
              
              
                I split the tar file into data_1.tar, data_2.tar, …, data_8.tar. For the k^{th} GPU, i.e., local_rank = k, the process read data_k.tar and build data loader with data_k.tar. Then I get 8 different data loaders (their data are different). In this case, I guess I should set shuffle=True and do not ne…