The pretrained models are most likely sticking to the literature for the corresponding model, which often used input images of the shape 224 x 224 (often randomly cropped to this shape).
Since these torchvision models use adaptive pooling layers the strict size restriction was relaxed and you would be able to pass bigger images and (some) smaller images. Note that the min. size depends on the conv and pooling operations, which would create an empty output, if your input is too small.
That being said, I would not assume to see a good performance for largely differently shapes input images and would try to finetune the model for this use case.