FastTokenizer add_special_tokens also adding individual characters for multi character tokens
# 🐛 Bug

## Information
Model I am using (Bert, XLNet ...): BERT FastTokenizer 

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [X] my own modified scripts: (give details below)

The tasks I am working on is:
* [X] my own task or dataset: (give details below)

As far as I can tell, adding tokens (e.g. '[EOS]') to the FastTokenizer  will result in all the single characters of the token to be added to the tokenizer ('E', 'O', 'S'). This seems to only occur when using tokenizer.add_special_tokens() as described [here](https://huggingface.co/transformers/main_classes/tokenizer.html#transformers.PreTrainedTokenizer.add_special_tokens). However, adding the tokens via the constructor seem to work just fine.
This bug results in problems using the uncased models as we don't want uppercase letters. Also the unwanted additions to the vocab don't seem to show up in len(tokenizer), which results in index of out range errors when feeding into BERT. 

This would be a breaking case: 
```  
    tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased', use_fast=True)
    print('Tokenizer len before: {}'.format(len(tokenizer)))
    num_added = tokenizer.add_special_tokens({'eos_token': '[EOS]', 'bos_token': '[BOS]'})
    print('Tokenizer len after: {}'.format(len(tokenizer)))
    print('Number tokens added: ', num_added)
    print(tokenizer.bos_token)
    print(tokenizer.eos_token)
    # We can see that the tokens have been added successfully 

    # However, encoding the same sequence as before, we run into problems:
    encoded = tokenizer.encode('This is a big S!')
    print(encoded)
    print(tokenizer.convert_ids_to_tokens(encoded))
    # If you look carefully, you can see that the 'S' in the sequence is not lowercase.
    # Also the id in the line above (30526) should not be higher than the tokenizer len (30524) 
    # If we feed this into bert (after model.resize_token_embeddings(len(tokenizer))) this will crash 
    with an index out of range exception.
```
Outputs: 
```
Tokenizer len before: 30522
Tokenizer len after: 30524
Number tokens added:  2
[BOS]
[EOS]
[101, 2023, 2003, 1037, 2502, 30526, 999, 102]
['[CLS]', 'this', 'is', 'a', 'big', 'S', '!', '[SEP]']
```

Edit: My proposed workaround of adding the special tokens via the constructor also does not work. The tokens are accessible via tokenizer.<eos/bos>_token but adding tokens this way does not change the number of tokens in the vocab. I.e. using len(tokenizer) doesn't reflect the newly added tokens.


## To reproduce

Steps to reproduce the behavior:
Please find this [colab notebook](https://colab.research.google.com/drive/1hMEr0gpbyGJCZvIzFB22eKlmb9I-vKuu?usp=sharing) investigating the bug 



## Expected behavior
add_special_tokens() of the fast tokenizer should behave the same as for the regular tokenizer: Only adding the full special tokens '[EOS]' and not also single characters of it. Furthermore, I would expect that if something is added, this would also be reflected in the length of the tokenizer.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 2.9.1
- Platform: Colab, linux
- Python version: 3.7
- PyTorch version (GPU?): 1.5, gpu
- Tensorflow version (GPU?):
- Using GPU in script?: tried both, no difference
- Using distributed or parallel set-up in script?: No

Thanks for the extensive bug report! I can confirm that you are correct and that the issue does not occur when using the slow option.

Pinging @n1t0 