Backward pass error during OWL-ViT finetuning
### System Info

Google Colab

### Who can help?

@ArthurZucker 
@amyeroberts

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

https://colab.research.google.com/drive/1GfspAHLGTMmzfNShAVebzM04s32q1LKb?usp=sharing

### Expected behavior

I was trying to finetune Owl-ViT, and I came across this backward pass error. I investigated and found out that the error is the operator "/=" which does an inplace operation. 
At lines:

- https://github.com/huggingface/transformers/blob/4e8929dcbb9040f54f52d898a600260f338ef54f/src/transformers/models/owlvit/modeling_owlvit.py#L1296
- https://github.com/huggingface/transformers/blob/4e8929dcbb9040f54f52d898a600260f338ef54f/src/transformers/models/owlvit/modeling_owlvit.py#L1297
@pasqualedem 

Would you like to open a PR to fix it ? 🤗 