OSError Directory not empty error in Trainer.py on checkpoint replacement
### System Info

```shell
- `transformers` version: 4.20.0.dev0
- Platform: Linux-5.13.0-30-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.6.0
- PyTorch version (GPU?): 1.10.1 (True)
- Tensorflow version (GPU?): 2.7.0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: 4
- Using distributed or parallel set-up in script?: deepspeed
```


### Who can help?

@sgugger 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Create txt file of sentences.  
Ran run_clm.py with following parameters:   
deepspeed --num_gpus=4 run_clm.py --deepspeed ds_config_gptj6b.json --model_name_or_path EleutherAI/gpt-j-6B --train_file Jesus_sayings.txt --do_train --fp16 --overwrite_cache --evaluation_strategy=steps --output_dir ~/gpt-j/finetuned --num_train_epochs 5 --eval_steps 1 --gradient_accumulation_steps 32 --per_device_train_batch_size 1 --use_fast_tokenizer False --learning_rate 5e-06 --warmup_steps 10 --save_total_limit 2 --save_steps 1 --save_strategy steps --tokenizer_name gpt2   

Error traceback:   
```
[INFO|modeling_utils.py:1546] 2022-05-15 18:25:49,903 >> Model weights saved in /home/ubuntu/gpt-j/finetuned/checkpoint-3/pytorch_model.bin
[INFO|tokenization_utils_base.py:2108] 2022-05-15 18:25:49,911 >> tokenizer config file saved in /home/ubuntu/gpt-j/finetuned/checkpoint-3/tokenizer_config.json
[INFO|tokenization_utils_base.py:2114] 2022-05-15 18:25:49,917 >> Special tokens file saved in /home/ubuntu/gpt-j/finetuned/checkpoint-3/special_tokens_map.json
[2022-05-15 18:26:00,522] [INFO] [engine.py:3177:save_16bit_model] Saving model weights to /home/ubuntu/gpt-j/finetuned/checkpoint-3/pytorch_model.bin
[2022-05-15 18:26:26,263] [INFO] [logging.py:69:log_dist] [Rank 0] Saving model checkpoint: /home/ubuntu/gpt-j/finetuned/checkpoint-3/global_step3/zero_pp_rank_0_mp_rank_00_model_states.pt
[2022-05-15 18:27:44,462] [INFO] [engine.py:3063:_save_zero_checkpoint] zero checkpoint saved /home/ubuntu/gpt-j/finetuned/checkpoint-3/global_step3/zero_pp_rank_0_mp_rank_00_optim_states.pt
[INFO|trainer.py:2424] 2022-05-15 18:27:46,523 >> Deleting older checkpoint [/home/ubuntu/gpt-j/finetuned/checkpoint-1] due to args.save_total_limit
Traceback (most recent call last):
  File "run_clm.py", line 575, in <module>
    main()
  File "run_clm.py", line 523, in main
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/transformers/trainer.py", line 1320, in train
    return inner_training_loop(
  File "/home/ubuntu/.local/lib/python3.8/site-packages/transformers/trainer.py", line 1634, in _inner_training_loop
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_eval)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/transformers/trainer.py", line 1805, in _maybe_log_save_evaluate
    self._save_checkpoint(model, trial, metrics=metrics)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/transformers/trainer.py", line 1964, in _save_checkpoint
    self._rotate_checkpoints(use_mtime=True, output_dir=run_dir)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/transformers/trainer.py", line 2425, in _rotate_checkpoints
    shutil.rmtree(checkpoint)
  File "/usr/lib/python3.8/shutil.py", line 718, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/usr/lib/python3.8/shutil.py", line 659, in _rmtree_safe_fd
    onerror(os.rmdir, fullname, sys.exc_info())
  File "/usr/lib/python3.8/shutil.py", line 657, in _rmtree_safe_fd
    os.rmdir(entry.name, dir_fd=topfd)
OSError: [Errno 39] Directory not empty: 'global_step1'
  4%|██▌                                                         | 3/70 [21:59<8:11:00, 439.71s/it]
[2022-05-15 18:27:50,264] [INFO] [launch.py:178:sigkill_handler] Killing subprocess 78507
[2022-05-15 18:27:50,265] [INFO] [launch.py:178:sigkill_handler] Killing subprocess 78508
[2022-05-15 18:27:50,265] [INFO] [launch.py:178:sigkill_handler] Killing subprocess 78509
[2022-05-15 18:27:50,266] [INFO] [launch.py:178:sigkill_handler] Killing subprocess 78510
[2022-05-15 18:27:50,267] [ERROR] [launch.py:184:sigkill_handler] ['/usr/bin/python3', '-u', 'run_clm.py', '--local_rank=3', '--deepspeed', 'ds_config_gptj6b.json', '--model_name_or_path', 'EleutherAI/gpt-j-6B', '--train_file', 'Jesus_sayings.txt', '--do_train', '--fp16', '--overwrite_cache', '--evaluation_strategy=steps', '--output_dir', '/home/ubuntu/gpt-j/finetuned', '--num_train_epochs', '5', '--eval_steps', '1', '--gradient_accumulation_steps', '32', '--per_device_train_batch_size', '1', '--use_fast_tokenizer', 'False', '--learning_rate', '5e-06', '--warmup_steps', '10', '--save_total_limit', '2', '--save_steps', '1', '--save_strategy', 'steps', '--tokenizer_name', 'gpt2'] exits with return code = 1
```


### Expected behavior

```shell
Should delete old checkpoint without error. 

Workaround:
Changed trainer.py line 2425 to

shutil.rmtree(checkpoint, ignore_errors=True)
```
This causes program to run without error but leaves behind ghost checkpoint directories with no content. Though these are gradually pruned.
```

Thanks for the report! That sounds like a reasonable fix. Do you want to make a PR with it?<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.<|||||>What's the status of this? Is there a workaround without editing the source?<|||||>No PR was raised to fix it, you should go ahead if you want to contribute :-)