Redundant normalisation of image and text features in OWL-ViT
### Who can help?

@alaradirik 

### Issue description

Hi,

Thank you for the codebase! As the title suggests, I think that in `modeling_owlvit.py` the image and text features are normalised twice while in the original codebase from Google Research they are normalised only once. In particular, in `modeling_owlvit.py` image and text features are normalised both in lines 1073-174 and in lines 1145-1146. On the contrary in the original code, in [https://github.com/google-research/scenic/blob/main/scenic/projects/owl_vit/layers.py](https://github.com/google-research/scenic/blob/main/scenic/projects/owl_vit/layers.py), the features are normalised only in lines 86-89 whereas in line 144 the normalisation parameter is set as `normalize=False` and there is a comment explicitly saying `Don't normalize image and text embeddings:`. 

I think this is sensible as there is no reason for double normalisation which normally leads to performance degredation. Please let me know what do you think, and whether I'm wrong as I might be missing something.
Hi @ekazakos, you're right but the image embeddings are not normalized twice. `OwlViTModel.forward()` is called within `OwlViTForObjectDetectiob.image_text_embedder()` with `return_base_image_embeds=True`. This assures that we can retrieve both the umodified CLIP (OwlViTModel) embeddings and logits using the normalized features and also the unnormalized image embeddings (lines 1085-1087).

The reason we do this is OwlViT is trained in two stages: (1) training CLIP / OwlViTModel without any modifications and (2) training the object detection head and fine-tuning the base CLIP model.

Hope this helps!<|||||>Hi @alaradirik,

I suspected that this was the reason you did that. And thanks for the clarification, it's helpful! Yet, that implies that the text features are indeed normalised twice, no?<|||||>> Hi @alaradirik,
> 
> I suspected that this was the reason you did that. And thanks for the clarification, it's helpful! Yet, that implies that the text features are indeed normalised twice, no?

Yes, you're right indeed and thanks for pointing it out! I'll be opening a fix PR shortly.<|||||>Glad I could help! Could you please let me know whether this boosts validation performance at all?
<|||||>
Hey @ekazakos, sorry for the delay! The issue will be fixed with this [PR](https://github.com/huggingface/transformers/pull/19712) but it doesn't affect the performance as double normalization yields the same results.
