My guess. I think it depends on how you compute the loss and backward.
– DataParallel: if you merge 2 batch at the end then compute a single loss = average loss over each example, and then do loss.backward(). Then summing is mathematically correct. 1 or 2 or more GPUs, the gradient computed this way should be the same
– DistributedDataParallel: if using 2 separate losses for each GPU: loss1 = average over example in batch1, and loss2 = average over example in batch 2. To simulate loss = average over examples = (loss1+loss2)/2. You can loss1.backward(), loss2back.ward(), then average params gradient, that is equivalent to loss.backward()