As for the register_hook function:
It stated that the hook will be called after the gradient is computed, this is true for CPU and GPU use the same stream to compute, but how about I use the grad in a new stream on GPU, is it possible the grad is not fully computed as stated in Horovod 13?