Yes, you would have to pad all the point clouds to the maximum point cloud size. Apart from the very process of padding, even operations on the padded sequence will be computationally more expensive as compared to doing it without padding. The cost would be proportional to the amount of padding required for each point cloud which indirectly depends on the size of the maximum point cloud.