Hi,
I was running the resnet 50 on DGX 4GPUs using pytorch imagenet examples. And I set the bucket_cap_mb=90 MB. So I guess the Backpropagation and All-Reduce will not be overlapped.


In my opinion, All-Reduce, Broadcast should launch similarly to all GPUs, but the visible Profile data is All-Reduce and Broadcast unsynced.


And Why two GPUs broadcast for a long time, And I want to know why GPUs call Broadcast.


image1066×767 165 KB
Thanks.