Reproducibility when using pretrained GPT2
# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

Ensure reproducibility when loading pretrained models.

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

In order to distribute results in a transparent manner, it is important to ensure reproducibility.

When loading a pretrained model, I cannot get the same weights, probably due to non-initialized weights.
Even when using `set_seed`, I cannot get twice the same models.

## Your contribution

<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->

I created a [small example](https://colab.research.google.com/gist/borisdayma/efa9ce5e8c7078bf12031b525f21f107/transformers-repeatability.ipynb) to illustrate the issue.
Definitely seems bad. Have you experienced this with models besides GPT2? 


I think there might be two issues here:
1) `trainer.utils.set_seed` doesn't work completely: @julien-c 

2) I'm not sure whether this warning is expected behavior:
```
Some weights of GPT2LMHeadModel were not initialized from the model checkpoint at gpt2 and are newly initialized: ['h.0.attn.masked_bias', 'h.1.attn.masked_bias', 'h.2.attn.masked_bias', 'h.3.attn.masked_bias', 'h.4.attn.masked_bias', 'h.5.attn.masked_bias', 'h.6.attn.masked_bias', 'h.7.attn.masked_bias', 'h.8.attn.masked_bias', 'h.9.attn.masked_bias', 'h.10.attn.masked_bias', 'h.11.attn.masked_bias', 'lm_head.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
```<|||||>However, hashes won't be same even for a cloned state dict. Try to run this script
```
from copy import deepcopy
from transformers import GPT2LMHeadModel
import torch
from hashlib import blake2b


def get_hash(file):
    return blake2b(open(file, 'rb').read()).hexdigest()


model1 = GPT2LMHeadModel.from_pretrained("gpt2")
sd1 = model1.state_dict()
sd2 = deepcopy(sd1)
with open("sd1", "wb") as f:
    torch.save(sd1, f)
with open("sd2", "wb") as f:
    torch.save(sd2, f)
assert get_hash("sd1") == get_hash("sd2")
```
I also tried to save just individual weights from state dicts, but in this case hashes were equal. So I'm really confused, maybe 
it's an issue with Pytorch saving mechanism.<|||||>@sshleifer - I don't really that this issue is related to the Trainer.

To answer this issue, I don't think there is a bug regarding reproducibility. We have multiple tests that verify:
1) That saving / loading a pretrained GPT2 model produces the exact same results => the model has the exact same weights and logic => the model results are reproducible, see:
https://github.com/huggingface/transformers/blob/9d37c56bab8f7f1f1aa0b65be039516072254e77/tests/test_modeling_common.py#L75

2) That loading a pretrained GPT2 model from the modelhub always produces the same results => ... => the model are reproducible, see: https://github.com/huggingface/transformers/blob/9d37c56bab8f7f1f1aa0b65be039516072254e77/tests/test_modeling_gpt2.py#L352

The second raised issue here are the warnings when loading the pretrained weights. There have been **a lot** of issues about this, e.g.: https://github.com/huggingface/transformers/issues/5800, https://github.com/huggingface/transformers/issues/5348, https://github.com/huggingface/transformers/issues/3553, https://github.com/huggingface/transformers/issues/5814 => These warnings happen because of two reasons:
1) It concerns the embedding matrix, where as the output embeddings are tied to the input embeddings so that the weights don't matter and are therefore not saved
2) It concerns these buffer weights: https://github.com/huggingface/transformers/blob/9d37c56bab8f7f1f1aa0b65be039516072254e77/src/transformers/modeling_gpt2.py#L128 , which are hardcoded values and don't need to be saved either.

Since we are getting a lot of duplicated issues about this it might be worth to spend some time to disable all of them (also pinging @LysandreJik, @sshleifer,  and @sgugger here in case you feel like taking a deeper look into the warnings :-)) <|||||>The issue may be more related to transparency than reproducibility.

It would be good to be able to prove which model was used at the start, and to show that 2 models are the same (even more if it's done for audit purposes in a late future).<|||||>Comparing hashes for saved model files does not work, as for models with **same** state dict hashes would differ (at least for GPT2). So maybe it's better to load state dicts into memory and compare tensors for each key.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
