How to figure out which pretrained tokenizers support emojis?
Hi, I am working on a dataset with emojis. I found that the BERT tokenizer doesn't support emojis, and we have to manually add them and train their embeddings (#7648). But the RoBERTa tokenizer seems to identify emojis on tokenization as the following code:
```Python3
tokenizer = AutoTokenizer.from_pretrained('distilroberta-base', use_fast=True, normalization=True)
tokenizer.encode('I 🙁 hate ☹️ to 😣 see 😖 this 😫 fail 😩 , 🥺 pls 😢 help 😭 me 😤')
```
outputs: 
``` 
[ 0, 100, 8103, 27, 10172, 4157, 42699, 9253, 12605, 7, 17841, 2469, 192, 17841, 25448, 42, 17841, 4958,
 5998, 17841, 15375, 2156, 8103, 8210, 3070, 2968, 29, 17841, 7258, 244, 17841, 12410, 162, 17841, 10470, 2]
 ```
None of this represents the "\<unk\>" token so, all of them should be trained embeddings. Is that right? Also, why are there weird characters in most of the words in the vocab of RoBERTa model like - 'ĸ', 'Ġthis', 'ĠðŁĺ', '«', 'Ġfail', 'ĠðŁĺ', etc.?
you can try this:
tokenizer.decode(tokenizer.encode('I 🙁 hate ☹️ to 😣 see 😖 this 😫 fail 😩 , 🥺 pls 😢 help 😭 me 😤'))

If the tokenizer successfully decodes back to origin emojis then yes! Your tokenizer can encode emojis.
In this case, you use distillroberta tokenizer which use BPE (Radford et al. 2019) method, hence your tokenizer can encode emojis.

![Selection_467](https://user-images.githubusercontent.com/42698038/125319165-81b07200-e308-11eb-947f-4169b1d8fa97.png)

<|||||>I already found this hack somewhere, but thanks anyway!