If you don’t want to resize or “fix” your input data in some way, you could alternatively use an adaptive pooling layer so that its output size will always be the specified one.
This would make sure that your model works with different input shapes.