In my opinion, All-Reduce, Broadcast should launch similarly to all GPUs, but the visible Profile data is All-Reduce and Broadcast unsynced.


The attached profile is not very clear so I’m not sure what sort of synchronization issue you’re referring to wrt to broadcast and all-reduce.


And Why two GPUs broadcast for a long time, And I want to know why GPUs call Broadcast.


There are a few scenarios where DDP performs a broadcast. First during initialization a broadcast is done to ensure all parameters on all nodes are the same before we start training. Secondly, if you’re using single process multi device mode (one process driving many GPUs) a broadcast is done to synchronize the parameters across multiple GPU devices.