Thanks for the update! Now I can see where the variable output shape is coming from.
Detection models work on candidates for each input sample and will filter out all detection results with a low score (or a hard candidate threshold). This would explain why the internal shape is a factor of the input size. This also explains why e.g. random noise inputs return empty predictions, where no segmentation/detection results could be found.
You would have to check the reference paper to verify this claim, but that’s at least my understanding of detection models.