@ekurtic I agree with your description of advantages of DDP in terms of communication, copy, sync etc. 
On the LR part, however, I am not sure. As per the discussion in this thread and few others, each gpu will individually process 4 images, loss will be calculated and then during backward function the gradients will be calculated individually first. Then DDP calculates the average (instead of sum) of all local gradients. This means that the gradient magnitude (the average value) is effectively in the near-range of that for batch size of 4, and not 32. So I am not sure whether LR of batchsize 4 should be applied or that of 32.