No you don’t have to do this. This should be an adaptive average pooling, which should pool always to the same spatial size (1x1 here) for all input sizes.
@Niki have you tried with 224x224 or 448x448 images? Theoretically the code provided by @ptrblck should work and this way we could make sure, the network is running fine and the problem lies within the image size. I don’t know if non-square images are a problem here but from my understanding they shouldn’t.