[OWL VIT] make daily CI happy
# What does this PR do?

Fixes a slow test for OWLViT while trying to integrate `accelerate` support for this model! 

cc @ydshieh @alaradirik 

Slow test that was failing: `tests/models/owlvit/test_modeling_owlvit.py::OwlViTModelIntegrationTest::test_inference_one_shot_object_detection`

all slow tests pass!

_The documentation is not available anymore as the PR was closed or merged._<|||||>Thanks @ydshieh , just updated the description! 
Yes without this change, the test would fail on GPU (`accelerate` tests excluded)