How to restore a training?
For example, I use "run_glue.py" to train a model and stop at Epoch 30, and how to restore the training process from Epoch 30?
You will have to modify the provided example to save/reload the model, optimizer and scheduler states.

I've updated the scheduler classes in #872 so that we can save/reload the schedulers with the standard PyTorch serialization practice:
```
torch.save(schedule.state_dict(), FILE_NAME)           # save
schedule.load_state_dict(torch.load(FILE_NAME))    # reload
```<|||||>Thanks for the response~<|||||>@thomwolf - I happen to need save/resume training for `run_glue.py`; I'm willing to implement this and make a PR if I can get feedback about the overall approach.

It looks like I would want to save:

- `global_step`
- `step` (from this one could presumably skip to the right place in the `epoch_iterator` of `train`)
- optimizer, model, scheduler (these look like they should be trivially amenable to `torch.save`)
- random seed, torch's random seed, numpy's random seed

Based on the current set up, I'm not actually sure what the right way to preserve the `train_sampler` and `train_dataloader` states are (without also serializing them too, which seems like a waste, but is by far the easiest way to handle it if they are so amenable), since their states can't be recreated with the above information. In my case, punting on these is an acceptable option, as well as being wasteful and saving to disk.

Thoughts?<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
