Urgent! Weird behavior of CLIPTokenizer when encoding out of vocabulary /non-English text with openai/clip-vit-base-patch32, and question about merges.txt.
### System Info

**Environment is Google colab**
- `transformers` version: 4.24.0
- Platform: Linux-5.10.133+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.15
- Huggingface_hub version: 0.10.1
- PyTorch version (GPU?): 1.12.1+cu113 (False)
- Tensorflow version (GPU?): 2.9.2 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@patil-suraj I've seen you answering questions about CLIP before and would really appreciate your help. Any other's prompt help would be deeply appreciated because I'm in the middle of a project.

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

The easiest way to reproduce is to open:[this colab notebook](https://colab.research.google.com/drive/1TjXIAWItblSGKFaCND9dYr7qrHoD4R4G#scrollTo=6zDhnjx9HjW1)
Step 1:
Download and import tokenizer

```
!pip install transformers
!pip install datasets
!pip install -qq  transformers ftfy
!pip install -qq "ipywidgets>=7,<8"
from transformers import CLIPTokenizer
!transformers-cli env
```

Step 2: Test how out-of-vocab tokens are actually tokenized
```
tokenizer = CLIPTokenizer.from_pretrained(
    "openai/clip-vit-base-patch32",
)
print("tokenizer vocab size:",len(tokenizer.encoder))
print("tokenize known sequence:",tokenizer.tokenize(" . "))
print("tokenize unknow sequence:",tokenizer.decode(tokenizer("谁 谁")['input_ids']))
print("Unknown token should be:",tokenizer.unk_token)
print("But tokenize() maps the unknown tokens to the tokens and values below")
tokenizer.tokenize("谁 谁"),tokenizer.convert_tokens_to_ids(tokenizer.tokenize("谁 谁"))
```

Step 3: Test abnormal behavior of adding space inside 
```
print("Inconsistent input and output: a space is added")
encoded=tokenizer.encode(tokenizer.tokenize("«è"))
print("Encoded:",encoded)#only has two values besides start and end tokens but decoded to 3 tokens?
print("Expected: <|startoftext|>«è <|endoftext|>. (no space between « and è)\nActual output:",tokenizer.decode(encoded))
```

Step 4: Test how this leads to being able to encode and decode a non-English sentence back while it should become unknown tokens. It's not useful anyway because now the decoded sentence has spaces added inside words.

```
input="«è cosa ormai risaputa che a uno scapolo in possesso di un'ingente fortuna manchi soltanto una moglie. questa verità è cosí radicata nella mente delle famiglie del luoho che, nel momento in cui un simile personaggio viene a far parte del vicinato, prima ancora di conoscere anche lontanamente i suoi desiderî in proposito, viene immediatamente considerato come proprietà legittima di una o l'altra delle loro figlie.»orgoglio e pregiudizio è uno dei primi romanzi di jane austen. la scrittrice lo iniziò a ventun anni; il libro, rifiutato da un editore londinese, rimase in un cassetto fino alla sua pubblicazione anonima nel 1813, e da allora è considerato tra i piú importanti romanzi della letteratura inglese. è la storia delle cinque sorelle bennet e dei loro corteggiatori, con al centro il romantico contrasto tra l'adorabile e capricciosa elizabeth e l'altezzoso darcy; lo spirito di osservazione implacabile e quasi cinico, lo studio arguto dei caratteri, la satira delle vanità e delle debolezze della vita domestica, fanno di questo romanzo una delle piú efficaci e indimenticabili commedie di costume del periodo regency inglese."
output=tokenizer.decode(tokenizer.encode(input))
print(output[:15],output[-14:])
output=output[15:-14]#removing start token and end token
print('Input: ',input)
print('Output:',output)

```

### Expected behavior

I'm dealing with a text dataset that is multilingual and wish to tell which sentence is non-English by counting the percentage of unk_tokens after tokenizing.

Based on my understanding, tokenizer.encode(string) is equivalent to tokenizer.convert_tokens_to_ids(tokenizer.tokenize(string)) and should map tokens that are not in the vocab to tokenizer.unk_token. Also spaces are ignored during encoding and decoding will add spaces between tokens. 

However this is not the case, tokenize() seems to map them to some token in merges.txt (https://huggingface.co/openai/clip-vit-base-patch32/resolve/main/merges.txt) and then map them into values according to vocab.json(https://huggingface.co/openai/clip-vit-base-patch32/resolve/main/vocab.json).
Sometimes this seems to separate the text further into sub-tokens according to merges.txt and thus add spaces between them when decoding. 

This behavior is very annoying because it treats non-English text and English text in the same way, decodes the sentence back to its original form, but randomly adds spaces inside these non-English words.

Thanks very much for your patience and help.
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.