So after suggestions, all of you, I am thinking about adding a scheduler with starting LR as 0.0001. However, I am skeptical about weight decay. My dataset is highly skewed, so i actually want my training to be difficult so that I dont fit quickly.