Hi @Hongkai_Zheng
Thank you!



 Hongkai_Zheng:

What DDP does is just to reduce gradient (synchronize over all devices) so that each replica of model see the same gradient


reducing gradient will be done by DDP automatically. if I print gradient on each GPU, the printed gradient should be identical. I saw some code call all_reduce function to sync loss. My question is:
if all_reduce is called manually and DDP also sync gradient, what will happen?
according to DDP tutorial, weights in each GPU should be identical, It means that DDP will sync initial weights(how to sync initial weights?) in each GPU to make sure all model in all GPU are identical, right?
if gradient is broadcast to each GPU, it means that all GPU run optimizer separately. For example, if we trained a model on 1 GPU with batch_size=32, lr = 0.001, should batch_size=32 and lr=0.001 when we train the model on 8 GPU?