Simply spoken:
The difference here is, that the kernels have been trained to a specific image size (224) and thus the features they usually extract are usually of a certain size.
If you use smaller images, the kernels might not be able to extract the features with the usual size, since they are smaller (ore larger), which may result in a difference in performance.
If your input size varies to much from 224x224 pixels the results may even become much worse.