CoNLL2003 ner_tags order mismatch between the dataset from HF and the pretrained model
@patrickvonplaten @dslim23 

@dslim23 's pretrained models such as:
https://huggingface.co/dslim/bert-base-NER
have the following NER tag order baked in:
`"O", "B-MISC", "I-MISC", "B-PER", "I-PER", "B-ORG", "I-ORG", "B-LOC", "I-LOC"`

while the https://huggingface.co/datasets/conll2003 dataset has:
`O (0), B-PER (1), I-PER (2), B-ORG (3), I-ORG (4) B-LOC (5), I-LOC (6) B-MISC (7), I-MISC (8)`

The mismatch leads to defunct accuracy measurements out of the box for the pretrained NER models; try, for instance:
`python examples/pytorch/token-classification/run_ner.py --model_name_or_path dslim/bert-base-NER --dataset_name conll2003 --output_dir /tmp/test-ner --do_eval`


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>@julien-c - another good example for HF Hub issue support haha<|||||>@vshampor - let's see what @dslim23 thinks :-)<|||||>@patrickvonplaten thanks for the ping – though in the case it's the script that should be able to remap labels no? The model looks correctly defined with https://huggingface.co/dslim/bert-base-NER/blob/main/config.json<|||||>(a bit like the roberta case that's hardcoded in example scripts when it should be supported automatically?)<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>Ok sorry only took a closer look now. I think what happened here is that the model was trained with an older version of `run_ner.py` where the `id2label` was not adapted to the dataset yet which then leads to problems when one only wants to evaluate the model. @sgugger - should we maybe add some logic like:

"If --do_train is False and config has id2label list then use that" instead of forcing to overwrite it depending on the dataset?<|||||>That logic is in the text classification examples but not the token classification examples indeed. Will try to fix this during the week (or did you want to do it?)<|||||>> That logic is in the text classification examples but not the token classification examples indeed. Will try to fix this during the week (or did you want to do it?)

Happy to take it over - I have some free time this week<|||||>Go ahead then :-)<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.