You could use the output or the activations from the penultimate layer and feed it to another classifier.
However, this won’t solve your issue of imbalanced classes.
If you don’t want to train another classifier on top of your models, you could also use the predictions and try to find a threshold where to use the one model over the other.