Best model not loaded in Trainer.hyperparameter_search
Hi,

I am inspecting the `hyperparameter_search` of `Trainer` class. It does not ignore the `patience` argument, however, it does ignore the `load_best_model_at_end`. This is a bug in my view, since `patience` should be ignored when `load_best_model_at_end` is unavailable. Attaching an example for `patience == 1`, `load_best_model_at_end == True` and `metric_for_best_model == "accuracy"`:
<img width="579" alt="Снимок экрана 2021-10-06 в 15 04 04" src="https://user-images.githubusercontent.com/36672861/136198585-a32588b6-ec1c-465f-b47b-5a2249bdbae2.png">
After this the training part is over, however, the reported final value is not the best one:
<img width="313" alt="Снимок экрана 2021-10-06 в 15 05 44" src="https://user-images.githubusercontent.com/36672861/136198804-49b8ba60-1876-48b6-9c2b-0dad5ae58b15.png">

I can make the merge request to solve the issue if necessary.


Hi @Aktsvigun  would mind sharing your notebook or py file. I think it can be fixed without PR?<|||||>@pratikchhapolika sure, please find the notebook attached. In this notebook have a look at run 17 (penultimate cell, penultimate run). On the first epoch, the accuracy equals 0.45, still the "best run" has the accuracy of 0.44.
[hyp_trainer_example.txt](https://github.com/huggingface/transformers/files/7295195/hyp_trainer_example.txt)

Github does not allow me to attach .ipynb file, hence I changed the extension with .txt. To open it, change its extension from .txt to .ipynb<|||||>Maybe of interest to @sgugger <|||||>Good afternoon! @sgugger any updates on this?<|||||>Yes the hyperparameter search framework ignore the `load_best_model_at_end` argument, and this is not something that can be changed as far as I can tell since they do not accept such an argument for the reporting. `load_best_model_at_end` should just not be used with HP-search, it's meant for standard trainings only.<|||||>See it, thanks! Closing then.