Make ZeroShotImageClassificationPipeline faster
# What does this PR do?

The pipeline makes separate calls to model for each candidate label. This commit combines all labels into one call.
Original code takes more that 60 seconds to process one image and 1000 candidate labels. Updated code takes less than 2 seconds.

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->
## Who can review?

Library:
- pipelines: @Narsil 
_The documentation is not available anymore as the PR was closed or merged._<|||||>Have you found `batch_size` argument which should take care of that ?

It's even better since with `batch_size` you can adjust the size of the batch independantly of the number of candidate labels, which makes it easier to adapt relative to hardware/model size.

And you can have batch_size=1000 with only 3 candidate labels, they really hare independant.<|||||>We tested batch_size argument, it doesn't work as expected and take long time and a lot of memory. 
In the pipeline `batch_size` separates candidate labels and runs one preprocess for each image/candidate_label pair. 
We expect batching happening for images and all candidate_labels for each image.
```
pipe = transformers.pipeline(
    task='zero-shot-image-classification',
    model='openai/clip-vit-large-patch14-336',
    framework='pt',
    device="cuda:0"
)

with open('labels.json', 'r') as f:
    l = f.read()
labels = json.loads(l)

res = pipe(images=['/home/user/cat_dog.jpg'], candidate_labels=labels[:250], batch_size=250)
```
It produces this:
```
OutOfMemoryError: CUDA out of memory. Tried to allocate 4.96 GiB (GPU 0; 10.76 GiB total capacity; 4.66 
GiB already allocated; 4.98 GiB free; 4.71 GiB reserved in total by PyTorch) If reserved memory is >> 
allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory 
Management and PYTORCH_CUDA_ALLOC_CONF
```

Running with this using main branch transformers took 56 seconds vs 2 seconds on fast-zero-shot-image.
```
res = pipe(images=['/home/user/cat_dog.jpg'], candidate_labels=labels[:1000], batch_size=100)
```




<|||||>_The documentation is not available anymore as the PR was closed or merged._<|||||>Ohh now I remember. 

I dove a bit deeper into the code.

The issue lies within `CLIP` itself and how it's working. There's essentially 2 batch sizes the image batch size, and text batch size.
And CLIP is returning an object being TEXT_BS * IMAGE_BS. Which means in this case we're doing a cross product of what we really need.

In addition to that, the current pipeline does batch the same image over and over (meaning it's going to pass several times in the vision encoder.

What we want in an ideal world, would be that the images get batched on their own, and get their representation encoded, and independently so do the `candidate_labels` (since we're also calculating them way too many times currently, once for each image in the pipeline.).

We **need** to keep `batch_size` functioning, which this PR currently silently breaks.

Now since this pipeline is only implemented for CLIP as of now, I think we can clean this up by breaking up the CLIP model into pieces. I'll try to figure out another solution.<|||||>I have created another PR with you as co-author to try and find a fix which could keep the performance you get here (potentially a bit better since I calculate candidate labels only once).

Would the other approach work for you ?

https://github.com/huggingface/transformers/pull/21897 <|||||>Closing in favor of #21897 