"zero-shot-image-classification"  pipeline with `VisionTextDualEncoderModel` needs manual feature_extractor and tokenizer input
### System Info

```shell
transformers: 4.20.1
platform: windows 11, google colab
```


### Who can help?

@Narsil

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```python
# works
from transformers import pipeline

pipe = pipeline("zero-shot-image-classification", model="openai/clip-vit-base-patch32")

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
pipe(images=url, candidate_labels=["a photo of one cat", "a photo of two cats"], hypothesis_template="{}")
```

```python
# error
from transformers import pipeline

pipe2 = pipeline("zero-shot-image-classification", model="Bingsu/vitB32_bert_ko_small_clip")

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
pipe2(images=url, candidate_labels=["고양이 한 마리", "고양이 두 마리"], hypothesis_template="{}")
```


```sh
TypeError                                 Traceback (most recent call last)
[<ipython-input-8-c1bcb0faaf45>](https://localhost:8080/#) in <module>()
----> 1 pipe2(images=url, candidate_labels=["고양이 한 마리", "고양이 두 마리"], hypothesis_template="{}")

3 frames
[/usr/local/lib/python3.7/dist-packages/transformers/pipelines/zero_shot_image_classification.py](https://localhost:8080/#) in preprocess(self, image, candidate_labels, hypothesis_template)
     90         for i, candidate_label in enumerate(candidate_labels):
     91             image = load_image(image)
---> 92             images = self.feature_extractor(images=[image], return_tensors=self.framework)
     93             sequence = hypothesis_template.format(candidate_label)
     94             inputs = self.tokenizer(sequence, return_tensors=self.framework)

TypeError: 'NoneType' object is not callable
```
[Colab](https://colab.research.google.com/drive/1CHrjJ7f7JcyMrEIcK18ieUHvS_1xKJKm?usp=sharing)

Currently I'm using it like this:
```python
from transformers import AutoModel, AutoProcessor, pipeline

model = AutoModel.from_pretrained("Bingsu/vitB32_bert_ko_small_clip")
processor = AutoProcessor.from_pretrained("Bingsu/vitB32_bert_ko_small_clip")
pipe = pipeline("zero-shot-image-classification", model=model, feature_extractor=processor.feature_extractor, tokenizer=processor.tokenizer)
```


### Expected behavior

work with `pipeline("zero-shot-image-classification", model="Bingsu/vitB32_bert_ko_small_clip")`

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.<|||||>hi sorry for long reply, didn't see this until today:

The model `https://huggingface.co/Bingsu/vitB32_bert_ko_small_clip`  is a VisionTextDualEncoder, but it's not defined within the `AutoFeatureExtractor` meta class (@NielsRogge ) so the pipeline doesn't know about it and cannot load the `feature_extractor` that's why passing it manually works.

Basically the issue lies in transformers when we added this model, it wasn't properly configured.

Cheers.<|||||>@NielsRogge I also cannot find a small testing model here: https://huggingface.co/hf-internal-testing

For this dual model, is that normal ?<|||||>> Basically the issue lies in transformers when we added this model, it wasn't properly configured.

This was an incorrect assumption on my end. Those types of models are more generic, so they don't provide and `AutoFeatureExtractor`/`AutoTokenizer` property, so it's **normal** for them to fail.

Will update the pipeline loading to make it work<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.<|||||>Fixed per #18392 