CI 🚀 even more
# What does this PR do?

A follow up of #25274:

- To reduce `torch_job` reaches `95%` RAM --> with this PR, it reaches only `82%`.
  - Also smaller RAM usage for: `tf_job`: `60%` | `flax_job`: `86%`
- Avoid the non-modeling files being tested redundantly
  - we save the timing for ~ 2 x 8 = 16 min.    

Now, all the jobs of the full suite CI runs < 10 minutes (except the new job `non_modeling_job`, but it takes ~2 min to restore the cache!)

<img width="206" alt="Screenshot 2023-08-03 081339" src="https://github.com/huggingface/transformers/assets/2521628/07a8b1b5-7521-4d8c-8d7e-11b176c427c4">

Well, request a review too quickly, sorry, but just a few tiny thing to fix ...<|||||>_The documentation is not available anymore as the PR was closed or merged._<|||||>OK, fair point. At least a (closed) PR is in the history for reference if we ever need it in  the future. Thanks!<|||||>(we will need to keep an eye on the `torch_job` if something strange happens - mostly hanging in a full run: likely an OOM and some workers are killed.)<|||||>We can then go back to 6 workers instead of 8 if it happens.