A few more observations:
I am trying to understand how DistributedSampler prevents scattering inputs among nodes when using DistributedDataParallel. According to this answer, imagenet example uses torch.utils.data.distributed.DistributedSampler to do that.
This line:
       # compute output
        output = model(input)

however, calls forward on DistributedDataParallel which does call scatter 9:
    def forward(self, *inputs, **kwargs):
        self.need_reduction = True
        inputs, kwargs = self.scatter(inputs, kwargs, self.device_ids)

So even though DistributedSampler has the data already chunked up for each replica, the batches will be still scattered, but they will be scattered among local devices, not among nodes! It took me some time to realize that device_ids are all local devices for each node and DistributedDataParallel does not scatter among nodes.