Fix checkpoint deletion
# What does this PR do?

As pointed out on the [forums](https://discuss.huggingface.co/t/checkpoint-missing-optimizer-pt-how-to-resume/6138) there is a problem in the way checkpoints are deleted currently when `save_total_limit` is set and `load_best_model_at_end` is True. Since the best checkpoint is switched with the last checkpoint, we end up deleting the last checkpoint instead of the oldest available one.

This PR fixes this issue and adds tests.