Thanks Thomas, this was my hunch.  So in fact we end up with a feature map of at least 1x1 which we then up sample from, and the size of this feature map is related to the input image size.  The question is then does it perform best with a certain size?  I have made 256x256 tiles of a 4000x4000 satellite patch, would I be better using a different tiles size?
On a side note, is 256 tiles of this size enough to train on?  I have seen a lot of examples using transfer learning and saying it is important to do, however I have 22 channels in my data, so I assume I can’t make use of the vgg16 pre trained model as that will be just 3?