Could you check, if your model is still overfitting, i.e. is it predicts just a single class?
If so, could you post the model code as well as your training routine, so that we could check if for some errors?