Hello,
I am facing a similar problem. I wanted to compare VGG16, Resnet34 and mobilenetV2 on the same data-set, I obtain a 90% accuracy while using VGG, with the feature extractor being frozen, but Resnet34 and MobillenetV2 tend not to converge at similar levels(- max accuracy was 80%) with feature extractor being frozen.
I use the same learning rates( staring at 1e-4) with annealing , regularizes(L2-1e-4 and 1e-5), and  Adam.
Should I try SGD or other parameters.