OWLViT - Allow text model to compute text embeddings only once
### Feature request

Hi again, 

Currently, the OWL-ViT pipeline works by having a list of queries **per example** (e.g. for classification problems with 10 classes there would be 10 queries for each image). I understand that the rationale behind this is to allow to train with multiple datasets simultaneously where each dataset has a different set of classes, and where images from different dataset may be in the batch at the same time.

Nevertheless, in many settings (in most actually), we train/test using a single training/test set. And calculating the text embedding multiple times causes redundant computations. So, could you provide functionality for allowing the text embedding to be calculated only once? By once I mean **one time for each of the queries that correspond to the dataset classes**, and this to be used for all examples in the batch rather than calculating it for each batch element. Thank you!

cc @alaradirik 
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.<|||||>This is a great feature request and can help in improving the usability of OWL-ViT. Please do consider.

cc @alaradirik <|||||>Hi @amitjena40, sorry for my late reply! 

We looked into this issue and while we think it would make it easier to use OWL-ViT at a larger scale, it is a big breaking change as it requires introducing new arguments and rearranging multiple components. 