huangdi:

Another harder problem for me is that, when there are too many 1D tensors with different lengths on each gpu, is there any method to gather them easier without a loop?


You can pad each tensor to the maximum size and then use allgather. If you don’t know the maximum size before hand, you can first perform an allgather to collect the size of each tensor and then calculate the max.