[`InstructBlip`] Fix instructblip slow test
# What does this PR do?

Fixes the current failing daily CI test: https://github.com/huggingface/transformers/actions/runs/5675853423/job/15381807774 let's make the daily CI happy!

Ran the test on the latest docker image and the test now pass with these values.

cc @sgugger @ydshieh 

_The documentation is not available anymore as the PR was closed or merged._<|||||>The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25171). All of your documentation changes will be reflected on that endpoint.