If you are overfitting on the majority class, you could use weighted sampling to trade the sensitivity for the specificity (in a binary use case).
Note that the accuracy for an imbalanced dataset might be misleading as described in the accuracy paradox 1.