Okay I solved the issue myself. In the forward function of the densenet one can see that an adaptive average pooling 2d is used. There, the sliding window for the pooling is calcuated in such a way that the result is (something x 1 x 1) no matter of the input size.
forward function of densenet:
    def forward(self, x):
        features = self.features(x)
        out = F.relu(features, inplace=True)
        out = F.adaptive_avg_pool2d(out, (1, 1))  # the magic happens here
        out = torch.flatten(out, 1)
        out = self.classifier(out)
        return out