Solved by sytelus in post #22 
              
              
                Just found DistributedSamplerWrapper from here. It allows you to wrap DistributedSampler on the top of existing sampler. Might be good feature to add in PyTorch!