load_best_model_at_end
### System Info

- `transformers` version: 4.23.0.dev0
- Platform: Linux-4.15.0-189-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.3
- Huggingface_hub version: 0.10.0
- PyTorch version (GPU?): 1.8.1+cu111 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: <yes

### Who can help?

@sgugger ,Hi, When I finish traning t5 model with args ```--load_best_model_at_end```, I found that it still have four checkpoints dir at output_dir(I set max_epoch=4),and there is also a ```pytorch_model.bin``` at the output file.  Like they organized like this way
```
output_dir
--pytorch_model_1.bin

--ckpt1_dir
----pytorch_model_2.bin

--ckpt2_dir
----pytorch_model_3.bin

--ckpt3_dir
----pytorch_model_4.bin

--ckpt4_dir
----pytorch_model_5.bin
```

Does it means that the pytorch_model_1.bin is the best model or I can say I can AutoModel.from_pretrained(''./output_dir) would be correct?

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

None

### Expected behavior

None
Please use the [forums](https://discuss.huggingface.co/) for such questions as we keep issues for bugs and feature requests only. As indicated by the name of the arg, the best model is the one loaded inside the Trainer. You should save it wherether you want for further use.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.