does tokenizer support emoji?
Hi, I have the code below and it always encodes emoji as "unk". Can someone tell me what I should do? Thanks

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', do_lower_case=True)
s =" 😃 hello how are you"
tokenizer.tokenize(s)
['[UNK]', 'hello', 'how', 'are', 'you']
Hi! The tokenizer you're using (`bert-base-uncased`) was not trained with emojis, therefore it cannot tokenize them correctly. You should add this token to the tokenizer vocabulary:
```py
from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', do_lower_case=True)
s =" 😃 hello how are you"

tokenizer.add_tokens("😃")
print(tokenizer.tokenize(s))

# ['😃', 'hello', 'how', 'are', 'you']
```

Please be aware that the model you're using should have its embedding matrix updated to include the embedding for the new token added. You can see the [documentation here](https://huggingface.co/transformers/internal/tokenization_utils.html?highlight=add_token#transformers.tokenization_utils_base.SpecialTokensMixin.add_tokens), here's how you should update your model embedding matrix:
```py
# Let's see how to increase the vocabulary of Bert model and tokenizer
tokenizer = BertTokenizerFast.from_pretrained('bert-base-uncased')
model = BertModel.from_pretrained('bert-base-uncased')

num_added_toks = tokenizer.add_tokens(['new_tok1', 'my_new-tok2'])
print('We have added', num_added_toks, 'tokens')
 # Notice: resize_token_embeddings expect to receive the full size of the new vocabulary, i.e., the length of the tokenizer.
model.resize_token_embeddings(len(tokenizer))
```<|||||>Thanks, @LysandreJik !  I have another question. When I train using tweets, since there is a lot of noise, a tweet like 'This is soooo good' would be a problem for BERT tokenizer cuz "soooo" is not in the vocabulary. Is there a method to add all of them? Right now I am thinking about a kinda ugly way, just use nltk tweettokenizer to process all tweets and add to vocab with words, emoji, etc that appear frequently<|||||>Hi @steveguang, sentences like `This is soooo good` actually won't be a problem for the BERT tokenizer, as it can decompose the word `soooo` in multiple tokens:

```py
>>> from transformers import BertTokenizer
>>> tokenizer = BertTokenizer.from_pretrained("bert-base-cased")
>>> tokenizer.tokenize("This is soooo good")
['This', 'is', 'so', '##oo', '##o', 'good']
```

However, when working with a dataset that seems to have a lot of unknown tokens, it is generally a good idea to identify the tokens that come up relatively often and to add them to your tokenizer. A good example would be the emojis mentioned above, as these are an important attribute to the meaning of the sentence.