[`Mpt`] Fix mpt slow test
# What does this PR do?

As per title, failure report here: https://github.com/huggingface/transformers/actions/runs/5675853423/job/15381788713

Probably an issue with libs I had when designing the tests (had torch +cu117 instead of cu118) - ran the tests on the latest docker image and they all pass

cc @sgugger @ydshieh   

_The documentation is not available anymore as the PR was closed or merged._