Is the “rest” predicting the same labels? I.e. is class0 referring to the same class in all 10 heads?
If so, how are you processing the output, if all heads predict different labels and e.g. classifier_0 predicts 0 valid labels?