I think it depends on your final use case besides the training impact.
I.e. given that your targets are much larger than the model output: how would you want to deal with this shape mismatch when this model is deployed?
Would you need to increase the spatial size of the model output before further utilizing these predictions or would the model output be fine and it could be used with its small size? Depending on this, I would try to adapt this into the training.