The parameters are stored in float32 using the automatic mixed-precision util. and thus also the gradients. NCCL thus communicates them in float32, too. If you are calling .half() on the model directly and thus apply a pure float16 training, NCCL should communicate in float16.