I’ve debugged my script line by line, and found that the allocated memory get doubled when torch.distributed.Reducer is instantiated in the constructor of DistributedDataParallel.
I think the reducer is a necessary component for DDP, because it sums up the result from all the device.
But I don’t know how the reducer works, so that I still can’t understand why the memory gets doubled.

Is it expected behavior that the reducer takes the additional memory as the local model takes?
Does the reducer take the addition memory only for the rank:0 device?
I mean the addition memory consumption would not occur in the rank:1 or rank:2??
I can’t check this because I have only one gpu.