I’m trying to get DistributedDataParallel to work on a code, using pytorch/fairseq 6 as a reference implementation. I’m finding the implementation there difficult to comprehend. I’ve opened an issue 7 for the same. Below is a (hopefully) complete relevant extract. The uncommented segment I’ve already got working and loss in converging.
    def train_step(self, sample):
        self.model.train()
        self._optimizer.zero_grad()
        sample = move_to(sample, self.device)
        loss, batch_sizes = self.model(sample)
        # 1: Is the below done implicitly
        #    seems to be missing in fairseq code.
        # all-gather([loss, batch_sizes])
        # loss = loss.sum()/batch_sizes.sum()
        loss.backward()
         # 2: Something similar to the following 
        #    exist. what is happening here?
        # for p in parameters-optimized:
        #      p.grad = p.grad*distributed_world_size/batch_sizes.sum()
        self._optimizer.step()
        return loss.item()

My concerns are:

Shouldn’t I be doing an all gather as indicated in code? Is this done implicitly?
What is happening in the second segment?