Solved by Anakin in post #4 
              
              
                Hey @mrshenli, Thanks for your reply. 
I solved this issue already.  Basically,  there are two methods (three, plus your idea:D). 
These ideas are mostly based on  DistributedSampler and DistributedDataParallel . Using the DistributedSampler, a subset of data can be loaded correctly in this process. …