The Pascal VOC2012 dataset consists of “natural images”, while it seems the ISBI dataset is a medical image set.
Did you try to fine-tune the model or did you just use the pre-trained model for prediction?
Since the dataset for pre-training and your use case differs so much, I would definitely fine-tune the model.
You can find atutorial on transfer learning here 25.