Based on the results above, I assume reduce has worked on GPU because my tensors are put on CUDA deivces, am I right? However, I noticed that all the ranks have participated in the reduce algo. Meaning that rank 1, 2, 3 tensor values have also changed when performing the reduction. It seems that the reduction algo is quite naive, if I have 4 processes, it would run 3 rounds.
1st round, add rank 3 to rank 2.
2nd round, add rank 2 to rank 1.
3rd round, add rank 1 to rank 0.
So when I print the end result, not only rank 0 has the desired reduced sum value, but also rank 1 ~ (world_size -2) has also changed the value. Is this the supposed result of reduction? I thought rank 1 ~ (world_size -2) doesn’t store the imtermediate results. 

Hi the documentation might need to be updated.
I ran the reduce operation using gloo and nccl.
gloo reduce runs
[75745] After reduction: rank 0 has data tensor([4.], device=‘cuda:0’), backend is gloo
[75747] After reduction: rank 2 has data tensor([2.], device=‘cuda:2’), backend is gloo
[75746] After reduction: rank 1 has data tensor([3.], device=‘cuda:1’), backend is gloo
[75748] After reduction: rank 3 has data tensor([1.], device=‘cuda:3’), backend is gloo
[76980] After reduction: rank 2 has data tensor([2.]), backend is gloo
[76981] After reduction: rank 3 has data tensor([1.]), backend is gloo
[76979] After reduction: rank 1 has data tensor([3.]), backend is gloo
[76978] After reduction: rank 0 has data tensor([4.]), backend is gloo
nccl reduce runs
[75986] After reduction: rank 1 has data tensor([1.], device=‘cuda:1’), backend is nccl
[75985] After reduction: rank 0 has data tensor([4.], device=‘cuda:0’), backend is nccl
[75987] After reduction: rank 2 has data tensor([1.], device=‘cuda:2’), backend is nccl
[75988] After reduction: rank 3 has data tensor([1.], device=‘cuda:3’), backend is nccl
The result of the gloo reduce operation is wrong. It should be as described in the tutorial. This is a known bug, and there is an open issue for it ProcessGroupGloo reduce produces wrong result · Issue #21480 · pytorch/pytorch · GitHub 2.