Everything CLIP related seems to break starting form transformers 4.28.0
### System Info

- `transformers` version: 4.28.0
- Platform: Linux-5.10.107+-x86_64-with-glibc2.31
- Python version: 3.9.16
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 1.11.0+cu113 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help?

@amyeroberts 

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

It seems to me that there is some regression starting from transformers 4.28.0 that affects the CLIP vision model and everything related to it. 

In particular, I am having issue with
* ClipSeg
* the CLIPVisionModel proper.

# ClipSeg

For ClipSeg, I am able to use it and get the expected masks, essentially by literally following the example [here](https://huggingface.co/docs/transformers/model_doc/clipseg#transformers.CLIPSegForImageSegmentation):

```python
from transformers import AutoProcessor, CLIPSegForImageSegmentation
from PIL import Image
import requests

processor = AutoProcessor.from_pretrained("CIDAS/clipseg-rd64-refined")
model = CLIPSegForImageSegmentation.from_pretrained("CIDAS/clipseg-rd64-refined")

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)
texts = ["a cat", "a remote", "a blanket"]
inputs = processor(text=texts, images=[image] * len(texts), padding=True, return_tensors="pt")

outputs = model(**inputs)

logits = outputs.logits
print(logits.shape)
```

Then `logits` contains the logits from which I can obtain a mask by something like

```python
mask = torch.exp(logits)
mask /= mask.max()
```

I tested this and it works reliably until transformers 4.27.4. But with transformers 4.28.0, I get masks that are completely black regardless of the input image.

# ClipVisionModel

This is harder to describe, since it relies on an internal model. I have trained a model that makes use of the image embeddings generated by ClipVisionModel for custom subject generation. Everything works well until transformers 4.27.4. If I switch to 4.28.0, the generated image changes completely. The only change is installing 4.28.0.


In fact, if I save the embeddings generated by CLIPVisionModel with the two different versions for any random image, I see that they are different. to be sure, this is how I generate image embeddings:

```python
clip = CLIPModel.from_pretrained(...)
preprocessor = CLIPProcessor.from_pretrained(...)
...
encoded_data = preprocessor(
    text=prompts,
    images=images,
    return_tensors="pt",
    max_length=77,
    padding="max_length",
    truncation=True,
)
clip_output = clip(
    input_ids=encoded_data.input_ids,
    pixel_values=encoded_data.pixel_values,
)
image_embeds =clip.visual_projection(
    clip_output.vision_model_output.last_hidden_state
)
```

For reference, I am using clip-vit-large-patch14

### Expected behavior

I would expect CLIPVisionModel to give the same result on the same image, both in 4.27.4 and in 4.28.0
Just to make something reproducible, here we can see that the output of CLIPProcessor changes. I run the script

```python
from PIL import Image
import requests

import transformers
from torchvision.transforms.functional import to_tensor
from transformers import CLIPProcessor

processor = CLIPProcessor.from_pretrained("openai/clip-vit-large-patch14")

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)
reference = to_tensor(image)

encoded_data = processor(
    text=[""],
    images=[reference],
    return_tensors="pt",
    max_length=77,
    padding="max_length",
    truncation=True,
)

print(transformers.__version__)
print(encoded_data.pixel_values.mean())
```

With 4.27.4 I get

```
4.27.4
tensor(0.2463)
```

With 4.28.0 I get

```
4.28.0
tensor(-1.6673)
```<|||||>I figured out the issue: the CLIPProcessor expects tensors in the range [0, 255], but only starting from transformers 4.28.0. This seems a pretty breaking change to me! If I multiply my tensor by 255, I get the right results<|||||>Hi,

Thanks for reporting. This seems related to https://github.com/huggingface/transformers/issues/23096 and may be caused by https://github.com/huggingface/transformers/pull/22458. cc @amyeroberts <|||||>Hi @andreaferretti, thanks for raising this issue! 

What's being observed, is actually a resolution of inconsistent behaviour of the previous CLIP feature extractors. I'll explain: 
* to_tensor() doesn't just convert to a pytorch tensor, it also rescales the values to be between 0 - 1
* The deprecated feature extractors and image processors use Pillow for resizing their images. 
* Pillow requires that for RGB, pixel values are uint8 between 0-255. 
* Therefore input images with float values are upscaled and cast to uint8 before being converted to a PIL.Image.Image

In the previous behaviour, images after resizing kept their upscaled values. Currently, if an image was upscaled during resizing, the pixel values are downscaled back e.g. to between 0-1. This ensures that the user can set `do_resize` to `True` or `False` and the only difference in the output image is its size (and interpolated pixels). Previously, if you set `do_resize=False`, then your image pixel values are never upscaled, they remain between 0-1, would be downscaled again, as is happening now. 

Rather than try to infer processor behaviour based on inputs, we keep the processing behaviour consistent and let the user explicitly control this. If you wish to input images with pixel values that have been downscaled, then you just need to tell the image processor not to do any additional scaling using the `do_rescale` flag:

```py
outputs = image_processor(images, do_rescale=False)
```

Alternatively, you could pass in the images without calling `to_tensor`. 

In the issues linked by @NielsRogge, this is also explained: https://github.com/huggingface/transformers/issues/23096#issuecomment-1557699476

However, this is the second time a similar issue has been raised, indicating that the behaviour is unexpected. I'll think about how to best address this with documentation or possible warning within the code. <|||||>Yeah, it would be useful to add a warning mentioning `do_rescale`, as well as mention this issue in the documentation of CLIP and related models