OWL-ViT training / fine-tuning code
### Feature request

Hi,

I've noticed that recently Google Research added the training and fine-tuning code for OWL-ViT in Scenic. Are you planning to integrate it in HuggingFace Transformers? Thank you!
cc @alaradirik <|||||>Hi @ekazakos, thanks for the suggestion! And yes, we are planning to integrate it to transformers shortly.

@sgugger @NielsRogge in the paper, authors first train a base CLIP model using an image size of 224 × 224 and then resize the image position embeddings with linear interpolation to 768 x 768 before fine-tuning the whole model on the object detection task. Is it a good idea to have separate sections in the configuration file for the inference and training modes?<|||||>Thank you so much!! <|||||>> in the paper, authors first train a base CLIP model using an image size of 224 × 224 and then resize the image position embeddings with linear interpolation to 768 x 768 before fine-tuning the whole model on the object detection task. Is it a good idea to have separate sections in the configuration file for the inference and training modes?

No I wouldn't do that, I'd just refer to [this script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/contrastive-image-text) if people are interested in training a CLIP model themselves. Afterwards, they can load the weights into the `OwlViTForObjectDetection` model using a conversion script. This conversion script should include the interpolation of the position embeddings.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.<|||||>May I ask for about how long will you release the training and/or finetuning code of OWL-ViT? I personally think this feature will boost the usage of the model very much.<|||||>@alaradirik yeah, we need the fine-tuning code :)<|||||>I also would like the fine-tuning code example.<|||||>If anyone is interested I have a repo here: https://github.com/stevebottos/owl-vit-object-detection which is based on the huggingface implementation. It's still a WIP though and more of an experiment, but it works. If @alaradirik is interested I can help get this in.