Fix poor past ci
# What does this PR do?

<img width="326" alt="Screenshot 2023-06-26 102340" src="https://github.com/huggingface/transformers/assets/2521628/461069ba-637a-4ebf-bb2e-103bba81bbcc">



:face_with_spiral_eyes: Let's be a bit nice to torch 1.11 and 1.10 🙏 .

(just a type issue: `(line 641)  RuntimeError: expected scalar type float but found double` introduced in #24334)
_The documentation is not available anymore as the PR was closed or merged._