Look at what convolutional layers and pooling layers do. They work on any input size, so your network will work on any input size, too.
You only have to change the fully connected layers such as nn.Linear in VGG. Newer nets use a GlobalPooling layer before the fully connected layers and you do not even have to change the last linear layer.