Sorry for the late response, a little busy. Actually I have searched for ‘inplace allreduce’ after you pointed out and try it with cpu tensors. This one-line-change indeed brings some improvement, but I haven’t made it CUDA compatible.  Thanks for your patch, I will try it and feedback as soon as possible!