Do you see train loss decreasing(and train accuracy increasing) while val loss increases(and val accuracy decreasing)?
In that case, your model is overfitting.

You can add more data
Try out more augmentations
Dropouts
Use a shallow net.