Bert variants pretrained on Wikipedia are easily downloaded. Are the optimizers from the pretraining also available?
Is there a pretrained optimizer checkpoint available that can be loaded in the same way as a pretrained model?

I noticed that though the pretrained models are available trained on Wikipedia (ex can load a pretrained distillbert using: <br/>`model = DistilBertForSequenceClassification.from_pretrained('distilbert-base-uncased'`)

But, I cannot find the optimizer from the end of the training run on wikipedia. There is no `checkpoint['optimizer']`

For my task, looking at optimizer internals (momentum, second moment, etc) from the end of training on wikipedia may be more useful to me than looking at optimizer internals from training on a downstream task (eg. GLUE). Does such a checkpoint exist (either for TF or Torch?)

 Environment info (not really relevant)
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.1.0
- Platform: Linux-4.15.0-123-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.7
- PyTorch version (GPU?): 1.6.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.
 
 albert, bert, GPT2, XLM: @LysandreJik 
 Trainer: @sgugger
 -->

## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)




This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.