Suggestion: exception handling for out-of-vocab in pretrained model
Dear concern,

I appreciate your favor for public implementation.
As you know, all NLP people have an interest in applying your gorgeous model to every NLP problem.

I am writing this to suggest to add exception handling or warning message about out-of-vocabulary when a pretrained model is used.

I had been suffered by an error message for two days with checking every my code, naturally, I had believed it is my programming problem again :)
Then I discovered that it is out-of-vocab problem.
The pretrained BERT model has vocab size 28996, but my dataset with BERT subword tokenizer has more than 30000 vocab.
Then it is an error, when the BERT model meets an index higher than 28996.

If we know the problem, a solution is quite simple, filtering or replacing it to UNK index.
Error messages are caused from somewhat embedding parts in CPU mode, but it was unclear to identify GPU mode, thus other people might be suffered like me.

I know the pretrained model is not designed for application, long documents with unusual vocabulary.
However, I believe many people will appreciate your favor if you add exception handling or warning message.
If you already have recognized the issue, please ignore this message :)


Best
Sungho
Hi Sungho,
This should already be taken care of by the BertTokenizer which defaults on the `unk` token when a sub-word is not in the vocabulary (see [here](https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/pytorch_pretrained_bert/tokenization.py#L372)).
What BERT model are you using?
Do you have a simple example you can share so I can try to reproduce the behavior?<|||||>Hi Tomas,

It is really shame to describe, I think it is just my typo problem again :(
Please ignore this issue.

You are absolutely right, there is an exception handling.
Just I loaded a wrong model in the BertModel after tokenizing.
"bert-base-cased" in the model part, Instead of "bert-base-uncased", which has a smaller vocab size than "uncased".

I am so sorry to waste your time, and appreciate your favor again! :)

Best
Sungho<|||||>Ok make sense, good to know.
So there is no problem in the end?<|||||>Yep, only my embarrassing here :)

Though there is no performance change, but I believe it is not a problem of pretrained model at all, but 
 just different application.

Thus, no problem for implementation at all, I appreciate it!<|||||>Ok, let's close the issue then. Feel free to open a new one if you have other problems.<|||||>May I know why cased vocabulary is smaller than uncased? 