Yes, this is expected currently, because DDP creates buckets to consolidate gradient communication. Checkout this 28 and  this 22. We could potentially mitigate this problem by setting param.grad to point to different offsets in the bucket so that we don’t need two copies of grads.