Fully compatible with the open clip Tokeniser
### Feature request

The open clip tokeniser has a ```pad_token_id``` of 0, but this cannot be achieved because ```CLIPTokeniser.__init__()``` cannot set a ```pad_token_id```

### Motivation

Related to https://github.com/huggingface/diffusers/issues/4153

Stable diffusion v2 and XL use the open clip tokeniser. To avoid increasing dependencies, the transformers must also have the same functionality.

### Your contribution

It seems possible to set pad_token_id directly, but it is not realistic.
```
tokenizer = CLIPTokenizer.from_pretrained("stabilityai/stable-diffusion-2-1", pad_token="<|endoftext|>")
tokenizer.pad_token_id = 0
```
@patrickvonplaten<|||||>cc @patil-suraj and @ArthurZucker <|||||>Hey, I am not really sure I understand the issue here. The `pad_token_id` is dependent on the `pad_token`. By default the `pad_token` is set to `'!'` which, maps to `0` : `tokenizer.decode(torch.tensor([0]))`, `tokenizer.convert_tokens_to_ids('!')`. If you set the padding token to a different value by hand, the following will happen: 
```python 

>>> tokenizer = CLIPTokenizer.from_pretrained("stabilityai/stable-diffusion-2-1", pad_token="<|endoftext|>")
>>> print(tokenizer.pad_token_id)
49407
>>> tokenizer.pad_token_id = 0
>>> print(tokenizer.pad_token)
 '!'
```<|||||>> Hey, I am not really sure I understand the issue here. The `pad_token_id` is dependent on the `pad_token`. By default the `pad_token` is set to `'!'` which, maps to `0` : `tokenizer.decode(torch.tensor([0]))`, `tokenizer.convert_tokens_to_ids('!')`. If you set the padding token to a different value by hand, the following will happen:

You are right, my implementation does not seem to be perfect. The purpose of this implementation is to match when converting from text to token_id. What I want to know is how to set pad_token_id to 0 without affecting the words that are normally used like ```!``` 


<|||||>You can't really do that unless you train a new tokenizer 😅 
You can add a new token, with a new index, which will prevent splitting `!`. The problem is that the embedding at position `0` might have been trained as padding token and is thus a random tensor (not updated by gradient computation). <|||||>cc @patil-suraj to explain the context around Stable Diffusion and OAI vs. openCLIP here maybe<|||||>Hey @laksjdjf , 

Indeed, there's a discrepancy between the `pad_token_id` in the open clip tokenizer and the `CLIPTokenizer` in `transformers`.  But we can't change it for the existing models for backward compatibility reasons.

But note that for the tokenizer used in SD2 and SDXL it's already set correctly cf https://huggingface.co/stabilityai/stable-diffusion-2-1/blob/main/tokenizer/special_tokens_map.json#L16

And a bit more context about padding token in CLIP. CLIP doesn't care about padding token and the wrong padding token will only affect inference when using all token embeddings (like Stable Diffusion). For training, even if the padding token is wrong (i.e if we use `eos` instead of `!`, it shouldn't affect)  because

-  `CLIP` did not use `attention_mask` during training.
- `CLIPTextEncoder` uses a casual mask, so the tokens to the right don't influence the hidden states of tokens to the left.
- `CLIP` is trained with contrastive loss, which is computed using the `projections`, and the `text_projection` is computed by pooling the `eos _token` embeddings, which will always be similar no matter what the padding token is, because `CLIPTextEncoder` is causal, so the eos embeddings won't be affected by tokens on the right.
- For downstream training (like SD), as long as a consistent token is used for padding, it shouldn't severely affect the training. But for inference, we will need to use the same token.

So the way CLIP is trained, it doesn't care about padding token. It'll only affect the inference if a different token (compared to the padding token used for training) is used for padding. And this is already taken care of in SD 2 and SDXL repos.<|||||>> But note that for the tokenizer used in SD2 and SDXL it's already set correctly cf https://huggingface.co/stabilityai/stable-diffusion-2-1/blob/main/tokenizer/special_tokens_map.json#L16

My concern is that the above process will result in ```!``` no longer being available in its normal sense.

<|||||>Hey @laksjdjf, 

As @patil-suraj mentioned CLIP never used a padding token for training. It was trained with a causal mask and only tokens **until the eos token** are taken into account when computing the CLIP contrastive loss. All tokens following the eos token have **no** influence on the model, so one could have added any token here. 

Now, it only matters if a pretrained CLIP is further fine-tuned as is done for SD. In this case the padding token was used to influence the loss and in that sense SD does not make a difference between `!` and a `padding` token. **But** this is purely due to the way SD uses CLIP for fine-tuning - this is not an inherit characteristic of CLIP.<|||||>Hmmm, maybe I just don't understand, but my question is about the behaviour of the ```! token```, not the behaviour of the ```pad token```. If  ```! token``` is converted to ```pad token```, it seems to  make a difference when processing text containing ```! token``` .

```Python console
>>>tokenizer = CLIPTokenizer.from_pretrained("stabilityai/stable-diffusion-xl-base-0.9", subfolder="tokenizer_2")
>>>prompt = "! !! !!!"
>>>input_ids = tokenizer(prompt,padding="max_length", max_length=tokenizer.model_max_length, truncation=True, return_tensors='pt').input_ids

>>>print(input_ids)
tensor([[49406,     0,     0,     0,     0,     0,     0, 49407,     0,     0, ...

>>>print(open_clip.tokenize(prompt))
tensor([[49406,   256,   748,   995, 49407,     0,     0,     0,     0,     0, ...
```