I don’t know, how the Xception model is implemented in Keras or what GlobalAveragePooling2D does (is it returning a single output scalar for each sample?).
However, if you are using adaptive pooling layers (which define the output size instead of the kernel size), your spatial input shapes would be flexible.
This line 11 shows the usage for the ResNet implementation.
Since PyTorch does not use any placeholder variables, you don’t have to set None for specific shapes.