After completion of Trainer.hyperparameter_search() attribute trainer.state.best_model_checkpoint references the last trained model instead of the best one
### System Info

- `transformers` version: 4.28.1
- Platform: Linux-5.19.0-41-generic-x86_64-with-glibc2.35
- Python version: 3.11.3
- Huggingface_hub version: 0.14.1
- Safetensors version: not installed
- PyTorch version (GPU?): 2.0.0+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no


### Who can help?

@sgugger

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

https://colab.research.google.com/drive/1Ht14ntTQy96-_zO-iVlwvAgkyY8t6vKc?usp=sharing

Call `Trainer.hyperparameter_search()`; when it completes, attribute `Trainer.state.best_model_checkpoint` and other `Trainer.state` attributes reference the last trained model, in the sequence of models trained by `Trainer.hyperparameter_search()`.

Note: to speed-up reproduction of the issue, I have limited the training dataset size in the provided code, line #49; that's why the evaluation metrics at the end of the hyperparameters search are poor.

### Expected behavior

After `Trainer.hyperparameter_search()` completes, attribute `Trainer.state.best_model_checkpoint` should contain the filename of the checkpoint with the **best** model among all the models trained during hyperparameters search, not the **last** model; that is, the model trained during the run indicated in the `BestRun` instance returned by `hyperparameter_search()`

Likewise, other `Trainer.state` attributes should relate to the same model, e.g:
`Trainer.state.best_metric`
`Trainer.state.epoch`
`Trainer.state.global_step`

Hyperparameter search does not play well with the best model indeed. That's not something in our roadmap for fixing, but we are happy to look at any PR!<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.