Basically the issue arises if your network is not fully convolutional, that is, if it consists of some Linear layers(Fully connected). Try removing those and you can probably get away with any input size you would like.