Save checkpoint asynchronously on cpu to keep GPU training going
### Feature request

I would like to save checkpoints during training as asynchronous
There is no need for training to wait the end of checkpoint saving

### Motivation

As Model size is bigger and bigger, Saving checkpoint during model training takes more time.
The important problem is that two jobs, saving checkpoint and model training, are synchronous; Model training wait until saving checkpoint will finish.
It causes ver expensive GPUs are idle under the process of saving checkpoint .

### Your contribution

I would be happy to submit a PR, might require some help.

Thanks!
Hi @ykihong0, thanks for raising this issue.

So that we can best understand the feature being discussed, is this checkpoint saving when using the `Trainer` class? <|||||>Hello @amyeroberts, Thanks for reply.

> So that we can best understand the feature being discussed, is this checkpoint saving when using the Trainer class?

-> Yes. I hope the above feature will be supported when the save_model method in Trainer class is called ( https://github.com/huggingface/transformers/blob/v4.27.2/src/transformers/trainer.py#L2718)

 <|||||>cc @sgugger <|||||>That's not something we have on our roadmap at the moment. Happy to look at a PR if someone wants to integrate this.<|||||>Thaks @amyeroberts @sgugger for reply
I will give a PR to integrate this feature later.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.