Your idea of using “specialists models” sounds interesting and I would suggest to dig a bit deeper into this approach.
As far as I understand you tried to train a separate model to classify the worst performing class against all others (stage0 model).
After this is done you are feeding this prediction into your base model (stage1 model) and try to classify all samples.
How are you feeding the prediction of your stage0 into stage1? It might be for example a scaling issue so that this particular useful feature will be difficult to learn as your other features might be completely in another range thus masking the prediction. Could you check it and rescale the features if necessary?
I’m not sure I understand the limitations of your dataset right. Could you provide some sample data with random values, e.g.:
nb_samples = 100
nb_featues = 10
seq_len = 45
nb_classes = 5
data = torch.randn(...)
target = torch.empty(..., dtype=torch.long).random_(nb_classes)

I would like to take a look, if a weighted sampling approach is not possible.