Hello，
I tried to reduce communication bottleneck by using auto mixed precision training. I also used torch.profiler to check the training performance.
In the timeline, I noticed that the all-reduce kernel is ncclAllReduceRingLLKernel_sum_f32.

Does it mean the fp16 gradients are casted to fp32 to be all-reduced?  Does something like ncclAllReduceRingLLKernel_sum_f16 exist?
My GPU is P100. PyTorch version is 1.9.1+cu111.