Hi, I am working on distributed.launch module recently, I have some question.

I think with the launch and distributedDataParallel(model), you don’t need to average grads manually.
2.During your training, does your gpu0 have more memory usage than the other gpus? I found that the other gpus have extra memory usage in gpu0, it’s annoying.