Trainer with save_total_limit=1 keeps 2 checkpoints when EarlyStoppingCallback is active
### System Info

Transformers: 4.21.3
Python: 3.10.4
OS: Linux  5.4.0-91-generic #102-Ubuntu SMP x86_64
Torch: 1.12.1+cu113
GPU: A100 40G


### Who can help?

@sgugger 

### Information

As the title says. See below for details.

### Tasks

I trained BERT-based sequence classifiers.

### Reproduction

1. Run a training () with `EarlyStoppingCallback` active, `save_total_limit` set to 1 and `early_stopping` to something (e.g. 3)
2. Run it for a number of iterations greater than `early_stopping`
3. Check the number of checkpoints.

What happens is that if the best performing checkpoint is not the last, both the last and the best checkpoints are kept.

### Expected behavior

With `save_total_limit=1`, I would expect that only the best checkpoint is kept. 
Thanks for the writeup. That's not a bug, but very much intended. This is the only instance in which the `save_total_limit` argument is not fully respected, because:
- we need to keep the best model
- we also need to keep the last model to be able to resume training if a crash happens<|||||>@sgugger "If a crash happens". But what about when the training has ended? The checkpoint is not really needed anymore then, is it?

In any case, I could not find anything about this exception in the documentation. If it is not because of my inaptitude at looking up information (which it well may be), it might be worth adding a sentence to the description of the `save_total_limit`.<|||||>Sure, do you want to make a PR for this?<|||||>Just to clarify: a PR for what? :smile: <|||||>To add the exception to the documentation and/or delete the checkpoint at the end of training.<|||||>@sgugger  I would like to fix this issue, can help on where this fix needs to go ? <|||||>@sgugger  I think we can close this ticket<|||||>Thank you guys for taking care of this!