What about the performance if you do not use a pretrained model?
If that’s all right, then the problem is probably about imput normalization
Please check the following post for more details.

  
    
    
    What's the range of the input value desired to use pretrained resnet152 and vgg19? vision
  
  
    input image is first loaded to range [0, 1] and then this normalization is applied to RGB image as described here: 



All pre-trained models expect input images normalized in the same way, i.e. mini-batches of 3-channel RGB images of shape (3 x H x W), where H and W are expected to be atleast 224. 
The images have to be loaded in to a range of [0, 1] and then normalized using mean=[0.485, 0.456, 0.406] and std=[0.229, 0.224, 0.225] 
An example of such normalization can be found in the imagenet …