@Hongkai_Zheng ,
Many thanks for your kind reply!



 Hongkai_Zheng:

It will slow down your code. You just sync the gradient twice but everything should remain the same.


If I broadcast loss or somehting to all GPUs manually,  the loss should be identical one the dist.all_reduce() is done. besides the function dist.all_reduce() should be called, what else should I do if I want to sync loss manually?



 Hongkai_Zheng:

I’m not sure what your question is in this example. But the second case (8 GPUs) in your example is equivalent to 1 GPU with batch_size= 32 * 8=256. So you may want to increase the learning rate since the batchsize is increased.


for this example, I am confused:
now that the average gradient is broadcast to each GPU and each GPU has its own optimizer and learning rate, why should we 8x learning rate?
Please check the code:
Line 259 of this link yolov3/train.py at master · ultralytics/yolov3 · GitHub 1
The upper code broadcast dataset to all GPUs. My question is: broadcasting dataset to all GPU manyally necessary? I assume this is done by dataloader automatically.