unexpected behavior on RoBERTa tokenizer when using additional special tokens 
## Environment info
- `transformers` version: 3.0.2
- Platform: Windows-10-10.0.18362-SP0
- Python version: 3.8.3
- PyTorch version (GPU?): 1.6.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: <no
- Using distributed or parallel set-up in script?: no

### Who can help
 tokenizers: @mfuntowicz 

## Information


When trying to tokenize using RoBERTa tokenized and a special token and using add_prefix_space=True, the token following the special token does not get a space. 

## To reproduce

Steps to reproduce the behavior:

1. run the following code

```
tokenizer = RobertaTokenizer.from_pretrained("roberta-base")
tokenizer.add_special_tokens('[d_s]')
print(tokenizer('[d_s] test', add_prefix_space=True))
print((tokenizer('test', add_prefix_space=True))
```

## output
{'input_ids': [0, 1296, 2], 'attention_mask': [1, 1, 1]}
{'input_ids': [0, 50271, 21959, 2], 'attention_mask': [1, 1, 1, 1]}


## Expected behavior
{'input_ids': [0, 1296, 2], 'attention_mask': [1, 1, 1]}
{'input_ids': [0, 50271, 1296, 2], 'attention_mask': [1, 1, 1, 1]}

The second token should not change because of the first one.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
