Are you using different learning rates for different parts of the model or did you run some experiments with the mentioned learning rates?
I’m not sure, which hyperparameters will work best for your use case.
I assume you are fine tuning a pretrained model currently or are you training from scratch?