Thanks for your help!
Here are all outputs & commands you mentioned:
from transformers import BertTokenizer
tokenizer = BertTokenizer.from_pretrained(‘bert-base-uncased’)
num = tokenizer.add_tokens([‘helloxxx’, ‘committed’])
tokenizer.tokenize(‘helloxxxcommitted’)
[‘helloxxx’, ‘committed’]
tokenizer.tokenize(‘hellocommitted’)
[‘hello’, ‘##com’, ‘##mit’, ‘##ted’]
tokenizer.tokenize(‘uncommitted’)
[‘un’, ‘##com’, ‘##mit’, ‘##ted’]
The reason it does not work is that the tokenizer already has the word commited, hence it does not add it, even though I’d like to add it as an additional substring.
By dealing with hashtags, I mean that I have words like this: #justnorthkoreathings and they get tokenized as: [’#’, ‘just’, ‘##nor’, ‘##th’, ‘##kor’, ‘##ea’, ‘##thing’, ‘##s’]. While I have a lot of confidence in BERT, I doubt this tokenization is a very good signal, as a) BERT is pretrained on Wikipedia articles & books where hashtags aren’t the norm b) I’d argue that attention is at its best, the fewer words it needs to combine - here it needs to combine 4 elements to understand north korea…
Would appreciate any tips, how to deal with this!