Trouble fine-tuning zero-shot image classification model
### System Info

transformers 4.30.2
python 3.11.4

### Who can help?

@amyeroberts @sgugger 

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I am trying to fine-tune a **zero-shot** image classifier, I am following this example for reference: https://github.com/huggingface/notebooks/blob/main/examples/image_classification.ipynb

What I changed from the notebook:

The checkpoint I am starting from: `laion/CLIP-ViT-L-14-DataComp.XL-s13B-b90K`

For the model I am using `AutoModelForZeroShotImageClassification` like this:

```
model = AutoModelForZeroShotImageClassification.from_pretrained(
    model_checkpoint, 
    label2id=label2id,
    id2label=id2label,
    ignore_mismatched_sizes = True,
)
```

When I run `trainer.train()`, I get this error:

> TypeError: CLIPModel.forward() got an unexpected keyword argument 'labels'

### Expected behavior

With my changes to the notebook, it should fine-tune for zero-shot pre-trained model `laion/CLIP-ViT-L-14-DataComp.XL-s13B-b90K`
Hi @moon001light, thanks for raising this issue.

Models that can be loaded with the `AutoXxx` API will have a shared input and output structure. In the example notebook, the model is loaded using `AutoModelForImageClassification`. Models loaded with `AutoModelForZeroShotImageClassification` have a different set of expected inputs, in particular, they don't accept `labels` and expect `input_ids`.  Here's a guide on performing the zero-shot image classification task using transformers: https://huggingface.co/docs/transformers/tasks/zero_shot_image_classification. 

Add more example scripts is on my to-do list. I'll make sure to include this task! <|||||>Thanks for the quick reply @amyeroberts , the link you gave seems to be a high level overview. Is there anything else besides replacing `labels` with `input_ids`? Could you point me to the code so that I can see the full list of expected inputs for `AutoModelForZeroShotImageClassification`? Appreciate your patience thanks :)<|||||>@moon001light If all you want is to train this particular checkpoint, then I would look directly at the architecture it loads, which in this [case is CLIPModel](https://github.com/huggingface/transformers/blob/ea91c2adca842da3d2f87e094504fa7d66a7008a/src/transformers/models/clip/modeling_clip.py#L1082). 

More generally, the auto groups are [defined in `modeling_auto.py`](https://github.com/huggingface/transformers/blob/main/src/transformers/models/auto/modeling_auto.py). For [AutoModelForZeroShotClassification](https://github.com/huggingface/transformers/blob/ea91c2adca842da3d2f87e094504fa7d66a7008a/src/transformers/models/auto/modeling_auto.py#L1256C7-L1256C46) the model architectures that can be used are listed under [MODEL_FOR_ZERO_SHOT_IMAGE_CLASSIFICATION_MAPPING_NAMES](https://github.com/huggingface/transformers/blob/ea91c2adca842da3d2f87e094504fa7d66a7008a/src/transformers/models/auto/modeling_auto.py#L980). 












<|||||>A bit of a newbie here, after looking into it I am still not sure how to properly set `input_ids`. This is what I tried:

```
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(model_checkpoint)

def collate_fn(examples):
    pixel_values = torch.stack([example["pixel_values"] for example in examples])
    input_ids = tokenizer(examples["label"], padding="max_length", truncation=True)
    return {"pixel_values": pixel_values, "input_ids": input_ids} 
```

But my `input_ids` is wrong :( Not really sure what `input_ids` should be. Looking forward to that example script of yours so I can learn @amyeroberts . Appreciate any help I can get :)<|||||>@moon001light I suggest inspecting the objects at each step to see what they are and contain. For example, the output of 

```python
tokenizer(examples["label"], padding="max_length", truncation=True)
```

is not `input_ids`, but rather a dictionary that contains `input_ids`.