Hey @khalil
Could you please provide details of the following question?


Which version of PyTorch are you using?


What’s the size of your model (how many GB)? You can get this by doing the following:
model_size = 0
for p in model.parameters():
    model_size += p.numel() * p.element_size()

Given the picture shown in the previous post, it looks like the model is about 27GB? If that is the case, then, yes, DDP would use another 27GB as comm buffers. We are working on improving this: https://github.com/pytorch/pytorch/issues/39022 8
I am curious how did you train this model locally without DDP? If the model is 27GB, after the backward pass, the grads will also consume 27GB. So local training without DDP will also use >54GB memory?
And if your optimizer uses any momentum etc., it’s likely the optimizer will also consume a few more X of 27GB. But looks like there are only 64GB memory available? Is it because your optimizer does not contain any states?


Does this getting worse when you use more nodes (e.g., scale from 2 nodes to 3 nodes)?