Add support for moving Trainer out of GPU memory
# 🚀 Feature request

Hello! I couldn't find any way of doing this, so I'm opening this issue. I'm sorry if there is already support for this and I couldn't find it.

It would be useful to have the ability to completely remove a Trainer and all associated objects (model, optimizer, etc..) out of the GPU memory without having to destroy the Trainer object. This should also allow the reverse operation, that is, once the Trainer is on CPU, it should be possible to move it back on GPU -- including all components and accounting for multi-GPU parallelism.

Essentially, it would be nice to have a `trainer.move_to_device()` function that took care of everything.

## Motivation

In my particular use case I have 2 relatively small GPUs and I would like to train multiple models in the same script in an interleaved fashion.  Something like:
``` python
# Ugly semi-pseudocode
while not done:
  for trainer in trainer_list:
    trainer.move_to_device(cuda)
    for epoch in range(epochs):  
      trainer.train()
    trainer.move_to_device(cpu)
```

I suspect it could also be useful for other scenarios where VRAM is limited and someone could want to train multiple models.

ps: I'm aware of `_move_model_to_device()`, but that seems to be limited to the model weights.

## Your contribution

I don't know much about how the Trainer handles device placement, but I'd be happy to test any proposed solution. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.<|||||>Sorry for not seeing this earlier! Pinging @sgugger for advice.<|||||>This might be useful indeed, but is not something I will have time to work on any time soon. Always happy to review a PR however!<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.