Directories not found when saving checkpoints
_rotate_checkpoints deletes some checkpoint directory which causes "directories not found" error
Do you mind making sure the code quality test runs before we merge? You can see how to do that in the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md)<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
