In this case, try to use data augmentation,  and use learning rate decay. However, I recommend you to share more details about your cureent setups (e.g., task, model, dataset properties, lr rate, etc). Its will be helpful to understand the problem and debug it.