Decode [UNK] from tokenizer
I am using Bert for NER and I have a problem when the tokenizer uses the token [UNK] to mask a word. How can I easily know which was the original work given its index?
@RodSernaPerez I am not sure what you are doing, but the model will do that if it doesn't find your word in the vocab.txt file. You have to use `BertTokenizer` before training, so that it splits all of the words that are not in the vocab file into word tokens. I think it will mark words that it wasn't able to tokenize with `[UNK]`. As to your question, what do you mean by index? I am assuming you know where the `[UNK]` was in white space tokenized word. You can concatenate BERT word tokens first and then you can try to find word marked with `[UNK]` in that list of concatenated tokens.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
