Flaky tests
Hi,
in this PR #12794 I somehow have flaky tests.

See the two commits from here
https://github.com/huggingface/transformers/pull/12794/commits/0ea13d6cf8a6a1ca5772f6a129bea9a7c06f5571
to here
https://github.com/huggingface/transformers/pull/12794/commits/e93eb12c01f8de9df6a1130041fe315372f9378b

The 2nd commit is empty and fails at an other location.
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.