Option to change Ray's gridsearch scope
# 🚀 Feature request

It would be great if we can get more control over how Ray is selecting the best trial in a hyperparameter search. Currently, the default value for [`get_best_trial`](https://docs.ray.io/en/latest/tune/api_docs/analysis.html#ray.tune.ExperimentAnalysis.get_best_trial) is used here:

https://github.com/huggingface/transformers/blob/7c5d79912a21880ce13d77881940458e90d98917/src/transformers/integrations.py#L299

namely `scope="last"`. So for each trial, it will simply take the _last_ checkpoint of that trial, and use its performance to compare with the other trials. This is not always ideal as it may very well be that some trials converge sooner than other and than overfit, leading to poor evaluation scores in their last checkpoints. Fortunately, Ray allows other options, such as `"all"`, which takes the best checkpoint instead of the last, of each trial and compares those.

## Motivation

Currently there is no way to pass this through to Ray from within `transformers` as I can tell, yet it is an important aspect of hyperparameter search.

## Your contribution

I can work on this if requested, although I am still looking for input how to best tackle this. One could simply add an argument to TrainingArguments, e.g. `ray_scope`, and then change this line

https://github.com/huggingface/transformers/blob/7c5d79912a21880ce13d77881940458e90d98917/src/transformers/integrations.py#L299

to

```python
best_trial = analysis.get_best_trial(metric="objective", mode=direction[:3], scope=trainer.args.ray_scope)
```

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.<|||||>Yes, I'd still like to see this added. I can do a quick PR if needed.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.<|||||>Another bump to remind myself that I'll do a PR for this in the coming month.