Current PyTorch core one splits the whole dataset in interleaved fashion:
  

      github.com
  

  
    pytorch/pytorch/blob/d5988c5eca0221e9ef58918e4f0b504940cb926a/torch/utils/data/distributed.py#L116



      
      
                  if padding_size <= len(indices):
                      indices += indices[:padding_size]
                  else:
                      indices += (indices * math.ceil(padding_size / len(indices)))[:padding_size]
              else:
                  # remove tail of data to make it evenly divisible.
                  indices = indices[:self.total_size]
              assert len(indices) == self.total_size
          
          
    # subsample
              indices = indices[self.rank:self.total_size:self.num_replicas]
              assert len(indices) == self.num_samples
          
          
    return iter(indices)
          
          
def __len__(self) -> int:
              return self.num_samples
          
          
def set_epoch(self, epoch: int) -> None:
              r"""
              Sets the epoch for this sampler. When :attr:`shuffle=True`, this ensures all replicas
      
    
  


  

  
    
    
  

  

indices = indices[self.rank:self.total_size:self.num_replicas]
Code in github.com/nvlabs/wetectron (which is derived from older version of Detectron2):
wetectron/distributed.py at 69436c0c6bb77e8dddb6f3f2b5d912f0d1ecd8fb · NVlabs/wetectron · GitHub
indices = indices[offset : offset + self.num_samples]
Why did older Detectron2 use a different way of sampling?