Solved by sgugger in post #3 
              
              
                As @mihai said, it saves the model currently inside the Trainer. If you call it after Trainer.train(), since load_best_model_at_end will have reloaded the best model, it will save the best model.