Problem with using pretrained BertTokenizer for Korean
I have a corpus that contains Korean sentences. Here is the output of a berttokenizer for a token:

`tok = BertTokenizer.from_pretrained('bert-base-multilingual-uncased')`
`tok.convert_tokens_to_ids(tok.tokenize('잘해놨습니다'))`
`[44628, 14840, 97071, 97089, 97104, 13212, 79427]`
`tok = BertTokenizer.from_pretrained('bert-base-multilingual-cased')`
`tok.convert_tokens_to_ids(tok.tokenize('잘해놨습니다'))`
`[100]`
transformers version: 2.4.1

Totally, 'cased' tokenizer produces more 'unknown' token than 'uncased' one for Korean. 
Is it a bug? 
Intuitively, I would say that this might not be a bug but a limitation of the size of the vocabulary. In the cased version, all data that the tokenizer is 'trained' on is cased, meaning that there are tokens in the vocabulary that only differs by case (e.g. `Be` and `be`). As a consequence, this may cause the vocabulary to be a lot bigger, leaving less room for other tokens. 

That is just my intuition and not based on any research.

You can try out KoBERT, though: https://github.com/SKTBrain/KoBERT<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
