I am wondering, during training, does gloo perform all_reduce automatically based on the tensor’s device type?

Yes, see https://github.com/pytorch/pytorch/blob/master/torch/lib/c10d/ProcessGroupGloo.cpp#L720 36. Essentially, we check the input’s device type, and run the appropriate operation based on that.

Also, when all_reduce is performed on GPU, does gloo fallback using nccl?

This doesn’t happen, the GLOO backend can be built with CUDA and supports GPU operations (https://github.com/facebookincubator/gloo/blob/master/docs/cuda.md 19)