@lewtun
When I say ‘same class’ I mean that it always predicts sci.space no matter what text I enter - even on a sports or religions text  even though this model should generate a MultiClass classification result…
Your suggestion with the
bert = TFAutoModel.from_pretrained(tranformersPreTrainedModelName, label2id=label2Index, id2label=index2label)
seems to change the situation in the way that the model predicts not just one class but about 3 on the 4 sample sentences. But it is still almost always wrong on the actual target even though I trained the model with a  seq_len=50 over 32 Epochs with these results:
loss: 0.0456 - accuracy: 0.9882 - val_loss: 0.1228 - val_accuracy: 0.9680
I am confused because I don’t understand why I have to supply the parameters since I am using the model without its in-build heads? I have used the exact same model configuration with a classic TensorFlow/Keras GloVe model and reached a worse accuracy but had better actual predictions - so, there still seems to be something strange here because the measured accuracy just does not add up with the actual prediction result