Hi, Thanks for your nice suggestion!
Another harder problem for me is that, when there are too many 1D tensors with different lengths on each gpu, is there any method to gather them easier without a loop?
Situation:
GPU 0: [torch.Tensor(101), torch.Tensor(102), torch.Tensor(103), ..., torch.Tensor(200)]
GPU 1: [torch.Tensor(201), torch.Tensor(202), torch.Tensor(203), ..., torch.Tensor(300)]
GPU 2: [torch.Tensor(301), torch.Tensor(302), torch.Tensor(303), ..., torch.Tensor(400)]
GPU 3: [torch.Tensor(401), torch.Tensor(402), torch.Tensor(403), ..., torch.Tensor(500)]

Result:
GPU 0: [torch.Tensor(101), torch.Tensor(102), ..., torch.Tensor(200), ...  torch.Tensor(201), ..., torch.Tensor(500)]
GPU 1: [torch.Tensor(101), torch.Tensor(102), ..., torch.Tensor(200), ...  torch.Tensor(201), ..., torch.Tensor(500)]
GPU 2: [torch.Tensor(101), torch.Tensor(102), ..., torch.Tensor(200), ...  torch.Tensor(201), ..., torch.Tensor(500)]
GPU 3: [torch.Tensor(101), torch.Tensor(102), ..., torch.Tensor(200), ...  torch.Tensor(201), ..., torch.Tensor(500)]

The order of tensors in the output list doesn’t matter.