Be nice to TF
# What does this PR do?

... and to @Rocketknight1 

To be serious: to avoid OOM issue introduced in #23234. Note `torch_job` use `pytest_num_workers=3`.

See [this comment](https://github.com/huggingface/transformers/pull/24071#issuecomment-1580778679). 
_The documentation is not available anymore as the PR was closed or merged._<|||||>LGTM! If the torch value is `3` we could probably reduce the TF value even lower, but let's try this first.<|||||>`7` avoids the issue, but still 96~98% memory. Change it to `6` and will merge.<|||||>The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_24076). All of your documentation changes will be reflected on that endpoint.