WIP: Added basic eos token based pooling
# What does this PR do?
This PR is still a WIP. This is based on [this issue](https://github.com/huggingface/transformers/issues/21029). The main problem is that when new tokens are added to the tokenizer and text model and learned such as with [textual inversion](https://textual-inversion.github.io/) the clip text model pools at the wrong location as the pooling is not done at the new token location and not the eos token id location

Fixes #21029


## Before submitting
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

Models:

- text models: @ArthurZucker and @younesbelkada
Current informal tests to confirm that behavior isn't changed: 

In original

```
from transformers import CLIPTokenizer, CLIPTextModel
tokenizer=CLIPTokenizer.from_pretrained("CompVis/stable-diffusion-v1-4", subfolder="tokenizer")
text_encoder=CLIPTextModel.from_pretrained("CompVis/stable-diffusion-v1-4", subfolder="text_encoder").to("cuda")
encoded = text_encoder(**tokenizer(["hi my name is bob"], padding=True, return_tensors="pt").to("cuda"))
encoded.pooler_output
```
gives
```
tensor([[ 4.5786e-01,  6.4382e-02,  6.3140e-01,  6.2113e-01,  9.6310e-01,
         -9.7793e-02, -7.2665e-01, -7.6867e-01,  6.6914e-02,  6.3608e-01,
          1.4786e+00, -1.9321e-01, -1.1228e+00,  1.8028e+00,  8.4215e-01,
         -5.4223e-01, -6.0589e-01,  1.1507e+00,  1.3731e-01,  8.6263e-01,
         ...
```<|||||>_The documentation is not available anymore as the PR was closed or merged._<|||||>Ok! Passed the first test and got the same result as above. The main problem I have with the current solution is we need to get the eos_token_id from the tokenizer like
```
encoded = text_encoder(**tokenizer(["hi my name is bob"], padding=True, return_tensors="pt", eos_token_id=tokenizer.eos_token_id).to("cuda"))
```
There might be ways to save eos_token_id in the transformer part but for now, I think this does fix the problem. Will make more tests in the future and improve quality.<|||||>Hey, quick tip, you can also initialize a class variable `eos_token_id` for example, which can be fetch in the `config` at `__init__` time. 
Now regarding the issue, I have to ask why not use the same technique as in `CLIP`? You mention pros and cons, would you mind giving more details? 
<|||||>@ArthurZucker Thanks for the tip! Will fix that asap.

And for the original implementation of CLIP, sorry let me clarify. I meant the original implementation of how CompVis/latent-diffusion trains on added tokens. In there, they force the placeholder token/added token to be a single token that is already within the embeddings during the textual inversion training. 
Pro: With this approach, it'll work for the current implementation of clip 
Con: It's a pretty strict requirement that the current textual inv script in diffusers doesn't have. 

But overall, I think this pr, once I fix things up, won't cause any problems to the existing implementations because we want to pool on the eos token anyway and it'll also end up working for the textual inversion scripts in diffusers which will be nice.<|||||>@ArthurZucker Hi! In the config, I noticed that the eos_token_id for the clip text model can be different from the tokenizer as follows
```
CLIPTextConfig {
  "_name_or_path": "CompVis/stable-diffusion-v1-4",
  "architectures": [
    "CLIPTextModel"
  ],
  "attention_dropout": 0.0,
  "bos_token_id": 0,
  "dropout": 0.0,
  "eos_token_id": 2,
  "hidden_act": "quick_gelu",
  "hidden_size": 768,
  "initializer_factor": 1.0,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "layer_norm_eps": 1e-05,
  "max_position_embeddings": 77,
  "model_type": "clip_text_model",
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "pad_token_id": 1,
  "projection_dim": 512,
  "torch_dtype": "float32",
  "transformers_version": "4.26.0.dev0",
  "vocab_size": 49408
}
```
is there some reason for this? I'll try testing out vocab_size-1(altho I don't think it's a good idea) for now<|||||>Will fix the checks.<|||||>Pulled from upstream so that some tests can pass. And changed a code bit and added documentation. I do think a better solution might be to fix the eos_token_id in the config. I'll try figuring out how to do that<|||||>Actually, this made the code diff a bit messy so closing this now and will make a new pr.<|||||>The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_21096). All of your documentation changes will be reflected on that endpoint.<|||||>Also the reason for the possible difference between the `model.config.xxx_token_id` and `tokenizer.config.xxx_token_id` is because they are not linked together. We usually make sure that they have the same value but nothing is forcing that. Biggest reason I see is dependency, and simplicity since you could use other tokenizer with the same model and vice versa. 😉 <|||||>Thanks for the comment! Will post q on the new pr just for other people who want to follow.