Have you tried to play around with the learning rate a bit?
The overfitting could also be due to an imbalanced dataset, but I’m not sure if that’s the case for your use case.
How many samples of each class do you have?