Hmm, but it seems that GLOO does support all_reduce op on CUDA tensors, right? So my question is why bother running the all_reduce on host (You have to cudaMemcpy the CUDA tensors to CPU and do the all_reduce there, and then transfer it back to GPU devices again)? Why not directly do all_reduce on GPU? It seems on GPU is faster, am I right?