How large is each image you pass to the model?
torchvision.models.vgg uses an adaptive pooling layer 2 which should take variable sized inputs. However, if you are using an older version, your model might be missing this layer.