Solved by pritamdamania87 in post #6 
              
              
                You can pad each tensor to the maximum size and then use allgather. If you don’t know the maximum size before hand, you can first perform an allgather to collect the size of each tensor and then calculate the max.