Hi @nielsr
I have now gone through your notebooks and have a further (slightly different) question on your choice of evaluation strategy. I thought I’d post it here to continue the conversation, even though it’s not exactly related to the title topic.
I have seen that you calculated the performance using the accuracy from sklearn.metrics.accuracy_score
However, the way in which you apply it in your code, i.e. accuracy_score(labels.cpu().numpy(), predictions.cpu().numpy())  looks like it’s doing mostly a token-by-token strict comparison, so if there are say 13 different labels, and the model get “B-ORG” instead of “I-ORG”, or “O” instead of “I-ORG” they would both be considered equally “wrong”.
I am reading here that there are many other ways to evaluate performance on NER datasets, which on the surface at least seem quite sensible, and look at the entities rather than at each individual token. Out of curiosity, is there a specific reason why you chosen token-level accuracy as opposed to, for example, entity-level F1 scores or other metrics?
Thanks a lot!