When I started unfreezing the layers of resnet the accuracy went from 50% to 83% using SGD optimizer, but it saturates at this stage. I am using data augmentation that was suggested for these type of images. Yes, I should try to spend more quality time on regularization. I have-not experimented with different optimizers yet, not sure if they would make a difference.