Not sure how the performance difference would be, but you could create an index tensor and use scatter_add_:
idx = torch.cat([torch.tensor([i]*s) for i, s in enumerate(sizes)])
torch.zeros(len(sizes)).scatter_add_(0, idx, X)