Ok, I see. I assume you are using transfer learning on the pretrained net?
If so, could you try to lower the learning rate or set a lower learning rate for the pretrained model and leave the high learning rate for the new layers. You can use the per-parameter options 6 from the optimizer.