You could try to create a staged classifier, where the first level would classify each sample into the “easy” and “hard” classes, and call the corresponding model.
However, you would of course have to retrain this model and would have to compare it to the standard approach of a single model.
I would guess that a single model with some tweaks (regularization, more aggressive data augmentation) would work better.