Solution:
Use the adaptive average pooling in the forward pass, where you have the shape information of the input available and can set the output shape to half the input shape.
In the original torchvision resnet50 implementation the adaptive average pooling layer is created during class initialization and I didn’t think of the idea that it doesn’t have to be this way.