nice to hear you got it working @dikster99! i had a quick look at your code and it looks good 
the reason the label → id and id → label mappings are needed is because the default behaviour is to create a dummy mapping based on the number of labels: transformers/configuration_utils.py at fa35cda91e9a1929f9cebeb973801709ba31dd4b · huggingface/transformers · GitHub 11
so in your case, the accuracy, loss etc were all fine because the model was using an internally-consistent definition for the labels, it’s just that the meaning of those labels did not match the ones in the data.