Sorry for my unclear description. I think you’re focus on the torch.gather method. And torch.cuda.comm.gather method is what I’m using which can communicate data between multiple GPUs and gather data to specific device.
Thanks
http://pytorch.org/docs/master/cuda.html?highlight=gather#torch.cuda.comm.gather 63