Thank you!
I’ve already made an issue on Github, Best model not loaded in Trainer.hyperparameter_search · Issue #13902 · huggingface/transformers · GitHub 5
There it’s shown that load_best_model_at_end is indeed ignored and probably need to be fixed.