Hi,

It will slow down your code. You just sync the gradient twice but everything should remain the same.
initial weights are synchronized when you construct DDP class. (DDP does it you don’t have to do it manually)
Right, each process has its own optimizer. But since the optimizer sees the same gradient, the update of the weights is also the same.
I’m not sure what your question is in this example. But the second case (8 GPUs) in your example is equivalent to 1 GPU with batch_size= 32 * 8=256. So you may want to increase the learning rate since the batchsize is increased.