A few things come to my mind:

Make sure your training dataset is balanced (not highly skewed towards one class).
Train for more epochs. Plot the loss function and see if the loss values are actually saturating to a very low value.
If Step 2. works fine, ensure you’re not overfitting using the validation set early stopping.