Checkpoints are saved multiple times during hyperparameter tuning / How to get the best model?
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.8.2
- Platform: Linux-4.19.0-16-amd64-x86_64-with-glibc2.10
- Python version: 3.8.0
- PyTorch version (GPU?): 1.9.0+cu102 (True)
- Tensorflow version (GPU?): 2.4.1 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no


### Who can help
Probably @amogkam because issue is related to ray tune.

## Information

Model I am using (Bert, XLNet ...): distilbert-base-uncased

The problem arises when using:
*  my own modified scripts: 

The tasks I am working on is:
* an official GLUE/SQUaD task: MRPC



My ultimate goal is to tune the hyperparameters of a model and save the best model in a folder.
Therefore I created the script below. It runs a simple hyperparameter search.

After the run, the folder `trainer_output_dir` is empty and the `ray_local_dir` folder has following structure:
````
ray_local_dir/
├── tune_transformer_pbt/
│     ├── _objective_081f1_00000_0_learning_rate=2.4982e-05_2021-07-13_14-44-15
│     │      ├── checkpoint_001377
│     │      │      ├── checkpoint-1377
│     │      ├── trainer_output_dir
│     │      │      ├── run-081f1_00000
│     │      │      │      ├── checkpoint-459
│     │      │      │      ├── checkpoint-918
│     │      │      │      ├── checkpoint-1377
````



When the attribute `save_strategy` (of the `TrainingArguments`) is set to `epoch` the folder structure like above is generated.
When the attribute `save_strategy` is set to `no`, then no checkpoints are written at all.
In folder `checkpoint_001377` the checkpoints are removed by ray tune but the checkpoints from the trainer_output_dir are not removed. The main reason is that the checkpoints are generated in two functions:
1.  in function [`_tune_save_checkpoint`](https://github.com/huggingface/transformers/blob/master/src/transformers/trainer.py#L892) which is called only when `self.control.should_save` is true
2. in function [`_save_checkpoint`](https://github.com/huggingface/transformers/blob/master/src/transformers/trainer.py#L1436) which is also called only when `self.control.should_save` is true

I care about these checkpoint folders, because this is the only way to load the best model.
[The example from ray tune](https://docs.ray.io/en/master/tune/examples/pbt_transformers.html) sets `load_best_model_at_end=True` but this has no effect and the trainer has no model which could be saved.
Thus I decided to load the model from the checkpoint folder.

The info in the BestRun object returned by `hyperparameter_search` is:
```
BestRun(run_id='081f1_00000', objective=0.5750986933708191, hyperparameters={'learning_rate': 2.49816047538945e-05})
```
It contains only the `run_id` and I search for the folder `_objective_081f1_00000.*` (glob pattern) in `tune_transformer_pbt` to get the right trials folder and then search for one folder which starts with `checkpoint`.
Is this the best way to load the model?

At the end of the notebook [How to fine-tune a model on text classification](https://github.com/huggingface/notebooks/blob/master/examples/text_classification.ipynb), it is stated:
`To reproduce the best training, just set the hyperparameters in your TrainingArgument before creating a Trainer:` but with PBT the hyperparameters might change during one trial. Furthermore, the model is already trained and it should not be necessary to train it once again.

Another question about the same topic:
Ray tune allows to set the scope for the function [`get_best_trial`](https://docs.ray.io/en/master/tune/api_docs/analysis.html#ray.tune.ExperimentAnalysis.get_best_trial) of the `ExperimentAnalysis` object which defines if the last result of a trial should be used to return the best one or if all intermediate evaluations should be also taken into account. At least what I can see in the [trainer class](https://github.com/huggingface/transformers/blob/master/src/transformers/integrations.py#L255), this parameter can not be modified. Due to the fact, that only one checkpoint per trial is saved (`keep_checkpoints_num=1`) and with the parameter `checkpoint_score_attr` we can also define the measure to compare, then it should be possible to only store the best checkpoint( according to the specified measure).
But the default value of `scope` is set to compare only the measure of the last result and it could happen that a trial's last measure is better even tough another trial had a much better result somewhen during the training (and exactly this checkpoint is kept on disk by `checkpoint_score_attr`) .
How do I need to configure hyperparameter search and training arguments such that I get the best model from all (intermediate) evaluations?



## To reproduce

Steps to reproduce the behavior:

Execute the following script:
```
from datasets import load_dataset, load_metric
from transformers import AutoTokenizer, AutoModelForSequenceClassification, Trainer, TrainingArguments
from ray import tune

model_name = 'distilbert-base-uncased'

tokenizer = AutoTokenizer.from_pretrained(model_name)
dataset = load_dataset('glue', 'mrpc')


def encode(examples):
    outputs = tokenizer(examples['sentence1'], examples['sentence2'], truncation=True)
    return outputs

encoded_dataset = dataset.map(encode, batched=True)

def model_init():
    return AutoModelForSequenceClassification.from_pretrained(model_name, return_dict=True)

def compute_metrics(eval_pred):
    metric = load_metric('glue', 'mrpc')
    predictions, labels = eval_pred
    predictions = predictions.argmax(axis=-1)
    return metric.compute(predictions=predictions, references=labels)

training_args = TrainingArguments(
    output_dir='./trainer_output_dir',
    skip_memory_metrics=True, # see https://github.com/huggingface/transformers/issues/11249
    disable_tqdm=True,
    do_eval=True,
    evaluation_strategy='epoch',
    save_strategy='epoch', # 'no', # TODO: change here to see the different behaviour
    logging_dir='./logs'
)

trainer = Trainer(
    args=training_args,
    tokenizer=tokenizer,
    train_dataset=encoded_dataset["train"],
    eval_dataset=encoded_dataset["validation"],
    model_init=model_init,
    compute_metrics=compute_metrics,
)

    
trainer.hyperparameter_search(
    direction="minimize",
    compute_objective=lambda x: x['eval_loss'],
    backend="ray", 
    n_trials=1,
    hp_space = lambda _: {
        #toy example
        "learning_rate": tune.uniform(1e-5, 5e-5),
    },
    scheduler=tune.schedulers.PopulationBasedTraining(
        time_attr="training_iteration",
        perturbation_interval=1,
        metric="objective",
        mode='min',
        hyperparam_mutations={
            "learning_rate": tune.uniform(1e-5, 5e-5),
        }
    ),
    keep_checkpoints_num=1,
    checkpoint_score_attr="training_iteration",
    resources_per_trial={"cpu": 1, "gpu": 1},
    local_dir="./ray_local_dir/",
    name="tune_transformer_pbt",
)
```


## Expected behavior

The checkpoints in the `trainer_output_dir` should not be written to disk.


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.