How to add new special token
I noticed the never_split functionality is no longer used to keep track of special tokens to never split on. If I wanted to add a new special token like '[NEW]' so the tokenizer never splits it, how should I go about doing that? (I've already manually added it to vocab.txt by replacing an unused token with [NEW]. Now I just need to not split it)
How did you add a new vocab.txt file? <|||||>I actually figured it out. I manually replaced one of the unused tokens in the vocab file with [NEW] and added "additiona_special_tokens": "[NEW]" to the special_tokens.json file in the same directory as the vocab.txt file. It works, but I realized that adding new tokens without the ability to do further pretraining isn't all that useful, especially given small dataset size. I decided not to do it.