run_tests_pipelines_torch is not deterministic
I am currently working on a PR and while making sure we pass all tests and checks, i ran into a pipeline test that failed in one run but succeeded afterwards with the same codebase (only a docstring has been changed). 

I think this nondeterministic test-behaviour should be avoided in an automatic build pipeline. 

Here it failed:
https://app.circleci.com/pipelines/github/huggingface/transformers/32972/workflows/ede22951-5fd9-446c-8417-09acd2caa979/jobs/347421

Here it worked:
https://app.circleci.com/pipelines/github/huggingface/transformers/33082/workflows/f297f4fa-bb81-417c-ae40-892747803400/jobs/348790

Feel free to request additional info, as I didn't know what else is needed.
Pinging @Narsil <|||||>Hi @kevinpl07 This PR should have fixed the issue. You test ran later, but were you rebased after this PR  https://github.com/huggingface/transformers/pull/15154 ?

I ran many tests on a local box without any issue on master just to be sure.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.