When I use torch.nn.parallel.DistributedDataParllel for training , I found process launched for  training on all other gpus would take up some memory at GPU 0 ,i.e worker 0. This take much more memory of GPU 0 that I can’t use bigger batch. How Can I solve this?  Is it because other process send gradients to 0 by default to calculate the average???
Screenshot_2019-04-10_18-11-05.png940×364 70.1 KB