Solved by farhadzadeh in post #6 
              
              
                Thanks to respond. However, I found out all_reduce, is for some reduceOP such as Max, SUM and so on. 
The one that I was looking for is torch.distribution.all.gather(list_tensor, tenso)