Solved by rvarm1 in post #4 
              
              
                Hi Nitin, 
To send all tensors to one GPU, you’d want to use dist.gather, which will gather all of the tensors onto one gpu (this is assuming you have one process running per gpu). If your tensor is t, then your call would look like: 
t = your_tensor(size)
if rank == 0:
# rank 0 is the node all tens…