Delete all checkpoints when set save_total_limit=1
### System Info

- `transformers` version: 4.31.0
- Platform: Linux-3.10.0-1160.el7.x86_64-x86_64-with-glibc2.27
- Python version: 3.11.4
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.1
- Accelerate version: 0.21.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@sgugger

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

1. Set the `save_total_limit` in `TrainingArguments` as 1 
2. Set `output_dir` in `TrainingArguments`
3. Run `trainer.train()`
4. No checkpoint folder was saved (all are deleted)

I have found the reason and fix the bug, just report this bug.
The reason is that in [trainer](https://github.com/huggingface/transformers/blob/main/src/transformers/trainer.py#L1963):
```
if self.args.should_save and self.state.best_model_checkpoint is not None and self.args.save_total_limit == 1:
            for checkpoint in checkpoints_sorted:
                if checkpoint != self.state.best_model_checkpoint:
                    logger.info(f"Deleting older checkpoint [{checkpoint}] due to args.save_total_limit")
                    shutil.rmtree(checkpoint)
```
This line:
```
if checkpoint != self.state.best_model_checkpoint:
```
Directly compare two paths as strings and ignore the format of the path.

For example, in my case, when:
* checkpoint == 'outputs/' 
and
* self.state.best_model_checkpoint == './outputs',

this comparison returns `False` and all checkpoints are deleted even including the best model.

I fix this bug by changing the above line to:
```
if str(Path(checkpoint)) != str(Path(self.state.best_model_checkpoint)):
```


### Expected behavior

The best model is saved and other checkpoints are deleted.
Hi @Pbihao ! 

Thank you a lot of reporting this issue. I can confirm it.

Would you like to open a PR to help us fixing this 🤗 ? <|||||>Yeah, I have submitted the PR #25136 .

Many thanks.