DistributedDataParallel in default uses extra buffer to synchronize gradients, thus consumes more memory (one extra copy of all grads) than what local training consumes.
In PyTorch 1.8, we added a new prototype flag “grad_as_bucket_view”, if you set it as True, it can save this extra copy of all grads, and should consume almost the same memory as local training.