[`IDEFICS`] Fix idefics ci
# What does this PR do?

Attempt to fix IDEFICS failing CI as discussed offline @stas00 - more info coming soon

original PR: https://github.com/huggingface/transformers/pull/24796

_The documentation is not available anymore as the PR was closed or merged._<|||||>The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25056). All of your documentation changes will be reflected on that endpoint.