Did you set the model to eval mode (model.eval())?
Usually the pre-trained models are in training mode after loading.
You can check this with model.training, which is a bool value.
Maybe some layers like BatchNorm have been updated?