Would DistributedDataParallel wrapper cost much GPU memory? In my case, the model cost around 7300MB when loaded into a GPU. However, when wrapped in DistributedDataParallel and run in the distributed mode, it costs 22000MB GPU momery.
Is it caused by the DistributedDataParallel wrapper? Are there any methods to save memory usage? Thanks!