Rotate checkpoint `shutil.rmtree(checkpoint)` fails
I was training `google/mt5-xl` model with `deepspeed` by huggingface trainer. The training was done in an aws `p3dn.24xlarge` node with 8V100 GPUs.
 
Trainer fails when [_rotate_checkpoints](https://github.com/huggingface/transformers/blob/7fcee113c163a95d1b125ef35dc49a0a1aa13a50/src/transformers/trainer.py#L1982) is called. Specifically in this [line](https://github.com/huggingface/transformers/blob/7fcee113c163a95d1b125ef35dc49a0a1aa13a50/src/transformers/trainer.py#L2005).
Apparently `shutil.rmtree` has this known [issue](https://github.com/ansible/ansible/issues/34335). 

Error Traceback:

```
Traceback (most recent call last):
  File "src/train.py", line 93, in <module>
    main()
  File "src/train.py", line 74, in main
    resume_from_checkpoint=checkpoint,
  File "transformers/src/transformers/trainer.py", line 1328, in train
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch)
  File "transformers/src/transformers/trainer.py", line 1409, in _maybe_log_save_evaluate
    self._save_checkpoint(model, trial, metrics=metrics)
  File "transformers/src/transformers/trainer.py", line 1528, in _save_checkpoint
    self._rotate_checkpoints(use_mtime=True, output_dir=run_dir)
  File "transformers/src/transformers/trainer.py", line 1954, in _rotate_checkpoints
    shutil.rmtree(checkpoint)
  File "/usr/lib/python3.6/shutil.py", line 490, in rmtree
    onerror(os.rmdir, path, sys.exc_info())
  File "/usr/lib/python3.6/shutil.py", line 488, in rmtree
    os.rmdir(path)
OSError: [Errno 39] Directory not empty: '/en-google_mt5-xl-1e-5-1234/checkpoint-320'
```


May be a `distributed barrier` is required for this to work properly.


@stas00 @sgugger
This code is only called from the main process (controlled by [this test](https://github.com/huggingface/transformers/blob/24cbf6bc5a0b6a9bb5afdda6bb1a329ac980fa4b/src/transformers/trainer.py#L1593)) so it's a not a distributed barrier issue.

Unless you are using AWS SageMaker with the model parallel extension, that `should_save` is only True for the local main process (or main process). Could you please give us more information to reproduce the bug?<|||||>I didn't use `AWS SageMaker`. Since you requested for more information, I started to think about a minimal example. The codebase I was working with is too large and contains redundant codes.
I took the script `run_summarization.py` and tried running with my environment. What happened that my transformers version was `4.6.0` from this [branch](https://github.com/huggingface/transformers/tree/t5-fp16-no-nans). I change my `tranformers` to `4.10.0.dev0` and the problem goes away. I could not reproduce the error.
Closing the issue for the time being. If I face the same error, I will open the issue again. <|||||>Ah, if you were on an older version, the barrier may not have been there, yes.