My best advice would be to try out some “reversely imbalanced” training runs, i.e. create more samples from the minority class in each batch, and try to force the model to overfit the minority class just for the sake of debugging.