Thanks very much! It is very clear.
Another question is that  when we train a model with DistributedDataparallel, DistributedSampler is suggested to use together.  If we need add sampler.set_epoch(epoch) before each epoch start according to set_epoch for DistributedSampler 41.