CLIP tokenizer inconsistent with OpenAI release
## Environment info

- `transformers` version: 4.6.1
- Platform: Linux-5.4.0-52-generic-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyTorch version (GPU?): 1.8.1 (True)
- Tensorflow version (GPU?): 2.4.1 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

@patil-suraj

## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [x] my own modified scripts: (give details below)
* [ ] the official example scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

```
>>> import clip
>>> import transformers
>>> clip.tokenize('hello world')
tensor([[49406,  3306,  1002, 49407,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0]])
>>> tokenizer = transformers.CLIPTokenizerFast.from_pretrained('openai/clip-vit-base-patch32')
>>> tokenizer('hello world')
{'input_ids': [3306, 220, 1002], 'attention_mask': [1, 1, 1]}
```

The HF CLIPTokenizer seems to add an extra token while dropping the <bos> and <eos> tokens. Am I missing something here?

Thanks!
The non-fast tokenizer seems to be fine:

```
>>> tokenizer = transformers.CLIPTokenizer.from_pretrained('openai/clip-vit-base-patch32')
>>> tokenizer('hello world')
{'input_ids': [49406, 3306, 1002, 49407], 'attention_mask': [1, 1, 1, 1]}
```<|||||>To add more into this, HF's fast tokenizer seems to add an extra token for every white space between words:
```
>>> tokenizer("a photo of a cat")['input_ids']
[320, 220, 1125, 220, 539, 220, 320, 220, 2368]
```<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>Hi, is there any update/eta on this?<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.