NER at the Inference Time
Apology in advance if this question might have already been asked. However, I have not been able to find a convincing answer or the optimum way to deal with this issue. 

To my understanding, NER makes a prediction at a token level. Since BERT is using a sub-word tokenizer, it is entirely possible that some part of the word won't get labeled or we have a different label within the same word. Both of these are undesirables because in the end we want the final result to be NER on a word, not token, level. 

For example, see [link](https://huggingface.co/dslim/bert-base-NER?text=Seven+of+the+men+are+from+so-called+%22red-flagged%22+countries%2C+including+Egypt%2C+Turkey%2C+Georgia%2C+Pakistan+and+Mali.Her+eighth+husband%2C+Rashid+Rajput%2C+was+deported+in+2006+to+his+native+Pakistan+after+an+investigation+by+the+Joint+Terrorism+Task+Force.If+convicted%2C+Barrientos+faces+up+to+four+years+in+prison.++Her+next+court+appearance+is+scheduled+for+May+18.). 

Barr(PER) ien(O) tos(PER)

This should have been Barr(PER) ien(PER) tos(PER). 

Another more confusing prediction [here](https://huggingface.co/dslim/bert-base-NER?text=27+Colors+for+Main+Fabric+available+in+singles+%26+together+in+Fatpack+Sweater)

F(MISC) abric (O) .... Fat (LOC) pack Sweater (ORG) 

We have inconsistent token production within the same words. 

So my questions are the following 

1) How can I best convert token-level NER labels to word-level labels? What is the best policy to deal with inconsistent token-level prediction within the same word? Is there a standard way to do this? Has this already been implemented in the huggingface library. 

2) Should not there be a way to force the model to recognize that those three tokens came from the same word so they need to have the same token-level label in the first place? 

Any suggestions are greatly appreciated. Thank you!

Hey @ndenStanford !
Thanks for the issue but I think the [forum](https://discuss.huggingface.co/) would be the best place to ask such general questions. We use issues to report bugs and for feature request. <|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.