XLM-RoBERTa can't add new tokens.
# 🐛 Bug

Model I am using (Bert, XLNet ...): XLM-RoBERTa

## To reproduce
Steps to reproduce the behavior:
 
1. tokenizer = XLMRobertaTokenizer.from_pretrained('xlm-roberta-large', do_lower_case=True)
2. tokenizer.add_tokens(['<a1>', '</a1>', '<a2>', '</a2>'])
3. tokenizer.convert_tokens_to_ids('<a1>')

It always respond 1 as ids for the new tokens.
The problem seems connected to: 
if (
                token != self.unk_token
                and self.convert_tokens_to_ids(token) == self.convert_tokens_to_ids(self.unk_token)
                and token not in to_add_tokens
            ):

in the second condition. 
When it tries to get self.convert_tokens_to_ids(token) always returns 1 instead of 3. 
3 is the id for unk_token.
1 is the id for pad.
Can you post a minimal verifiable example that we can just copy-and-paste to try? I guess you don't really try to add empty strings as tokens?<|||||>Hi @BramVanroy, no ofc, probably i've pasted the wrong snippet of code. 
If you try to expand the vocabulary of the tokenizer with the following code: 

```
tokenizer = XLMRobertaTokenizer.from_pretrained('xlm-roberta-large', do_lower_case=True)
tokenizer.add_tokens(['[A1]', '[/A1]', '[A2]', '[/A2]']) 
```

the size of tokenizer remains the same. 
Obviously if you try the same identical code with Bert or DistilBert (the ones i'm testing) all works fine. 
All seems connected with the second condition of if-else block: 

```
if (
   token != self.unk_token
   and self.convert_tokens_to_ids(token) == self.convert_tokens_to_ids(self.unk_token)
   and token not in to_add_tokens
):
```
This condition seems returns the wrong id for self.unk_token.
Removing this condition let me add the new tokens and extend the tokenizers.
<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
