@fmassa I am reading here 10  you have added a new DistributedSampler that also has shuffle. Why not just pass sampler=DistributedSampler(RandomSampler(dataset)) to a DataLoader?
I want to see if there is any particular issue with above.