Thank you for detail explanation.
Also, DDP and LR relationship are interesting. I used to find the LR with trial and error manner…
I got some understand about reduce and DDP. Please check my understanding.
So Basically DP and DDP do not directly change the weight “but it is a different way to calculate the gradient in multi GPU conditions”. If this is incorrect please let me know.
The input data goes through the network, and loss calculate based on output and ground truth.
During this loss calculation, DP or DDP work differently.
However I thought that gradient is basically calculated from loss.
Each loss in the GPU has the different loss result.
DP used mean value because DP send every output result to main GPU and calculate the loss.
If my understanding is incorrect please point out.
However DDP used the different. I still do not get it this parts. In the paper they also use the average value.
What is different between mean calculation and syncronized calculation?
For update the weight in network, the optimizer updates the network using by gradient value.
The update part is the optimizer part no DP or DDP related with it.
So the performance difference might come from LR difference? Because the bath size become different.  weight = previous weight - (gradient*learning_rate)
Really thank you for helping me.