Martin_Castin:

ncclAllReduceRingLLKernel_sum_f32


If the cost is dominated by allreduce communication, can you try no_sync context manager 1 to reduce the sync frequency?
Moreover, you probably can try to register a FP16 compression communication hook 6 to compress the gradients before allreduce. It’s one-line code change. You can try even more advanced gradient compression if interested.
I am really surprised at such a high allreduce cost.