Solved by Yanli_Zhao in post #8 
              
              
                in default, gloo is using ring algorithm gloo/allreduce.cc at master · facebookincubator/gloo · GitHub.  Gloo will do the all reduce on host, but NCCL will do the all reduce on devices