In the forward pass of applying a pre-trained model from torchvision to classify an image.
The zero padding in the convolutional layer will lead to that the values of boundaries positions of the feature maps become much large than the values on other positions.
In other words, the classification score mainly depends on the boundary pixels of images instead of object in the images. In other pre-trained models, e.g., from Keras, it is not the case.
What do you think about it? Am I wrong?