Hi.
I’m trying to use DistributedDataParallel in a single GPU node, for practice.
I checked my model’s size right after initializing it by
sum([p.numel()*4 for p in model.parameters()])/1024/1024, and it says 1140.xxx
and then I wrapped the model with DistributedDataParallel by
dmodel =DistributedDataParallel(module=model, ...).
then I checkout memory allocated by,
torch.cuda.memory_allocated()/1024/1024, and it says 2281.xx which is almost double of my model size.
I thought it’s due to using variable dmodel different from model.
So I retried to wrapping the model with DistributedDataParallel without using variable dmodel.
model = DistributedDataParallel(module=model, ...).
but torch.cuda.memory_allocated()/1024/1024 still says 2281.xx.
Is it expected behavior that memory consumption for model is doubled when using DDP?