MiniLM transformers inconsistent log posteriors in multiple runs
# 🐛 Bug

## Information

**Describe the bug**
Using MiniLM for computing log likelihood of test sentences. Cross posted [here](https://github.com/microsoft/unilm/issues/196)

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (attached below)

**To Reproduce**
Steps to reproduce the behavior:
1. pip install transformers==2.11.0, torch==1.5.0
2. Run the scripts pasted below, `hugging-face-bug-report.py`
3. Compare results across `gpt2`, `distilgpt2`, `microsoft/MiniLM-L12-H384-uncased`, `microsoft/DialoGPT-small`

**Expected behavior**
Log posteriors should not be different across multiple runs of the same model. 

Example run with `gpt2` [consistent]

`python hugging-face-bug-report.py -m gpt2`
```
Starting gpt2 on cpu [if available]
-19.95 Hello, my dog is cute
-20.09 Hello, your dog is cute
-25.92 Nothing is what everything isn't
```
`python hugging-face-bug-report.py -m gpt2`
```
Starting gpt2 on cpu [if available]
-19.95 Hello, my dog is cute
-20.09 Hello, your dog is cute
-25.92 Nothing is what everything isn't
```

Example run with `microsoft/DialoGPT-small` [consistent]

`python hugging-face-bug-report.py -m microsoft/DialoGPT-small`
```
Starting microsoft/DialoGPT-small on cpu [if available]
-37.22 Hello, my dog is cute
-31.38 Hello, your dog is cute
-31.30 Nothing is what everything isn't
```
`python hugging-face-bug-report.py -m microsoft/DialoGPT-small`
```
Starting microsoft/DialoGPT-small on cpu [if available]
-37.22 Hello, my dog is cute
-31.38 Hello, your dog is cute
-31.30 Nothing is what everything isn't
```
**BUT** Example run with `microsoft/MiniLM-L12-H384-uncased` [**inconsistent**]

`python hugging-face-bug-report.py -m microsoft/MiniLM-L12-H384-uncased`
```
Starting microsoft/MiniLM-L12-H384-uncased on cpu [if available]
-82.84 Hello, my dog is cute
-81.92 Hello, your dog is cute
-90.66 Nothing is what everything isn't
```
`python hugging-face-bug-report.py -m microsoft/MiniLM-L12-H384-uncased`
```
Starting microsoft/MiniLM-L12-H384-uncased on cpu [if available]
-78.01 Hello, my dog is cute
-75.90 Hello, your dog is cute
-83.02 Nothing is what everything isn't
```

- `transformers` version: 2.11.0
- Platform: macOS
- Python version: Python 3.6.10 :: Anaconda, Inc.
- PyTorch version (GPU?): 1.5.0 , No GPU
- Tensorflow version (GPU?): NA
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No
## Script 

```
# hugging-face-bug-report.py
#!/usr/bin/env python3

import torch
import argparse
from transformers import AutoTokenizer, AutoModelWithLMHead


LABEL_FIELD_DICT = {'gpt2': 'labels',
                    'distilgpt2': 'labels',
                    'microsoft/MiniLM-L12-H384-uncased': 'lm_labels',
                    'microsoft/DialoGPT-small': 'labels'}


class LM(object):

  def __init__(self, model_name='gpt2', device='cpu'):

    print('Starting {} on {} [if available]'.format(model_name, device))

    self.model_name = model_name
    self.device = torch.device(device if torch.cuda.is_available() else 'cpu')

    self.tokenizer = AutoTokenizer.from_pretrained(model_name)
    self.model = AutoModelWithLMHead.from_pretrained(model_name).to(self.device)

  def prepare_batch(self, texts):
    tokenized_input = []
    for text in texts:
      text_ids = self.tokenizer.encode(text, add_special_tokens=True)
      tokenized_input.append(text_ids)
    lens = list(map(len, tokenized_input))
    maxlen = max(lens)
    for i, t in enumerate(tokenized_input):
      tokenized_input[i] += [self.tokenizer.unk_token_id] * (maxlen - len(t))
    return torch.tensor(tokenized_input), torch.tensor(lens)

  def score(self, texts):
    with torch.no_grad():
      tensor_input, lens = self.prepare_batch(texts)
      mask = torch.arange(tensor_input.size(1))[None, :] < lens[:, None]
      labels = tensor_input.clone().detach()
      labels[~mask] = -100
      params = list(map(lambda x: x.to(self.device),
                        [tensor_input, mask, labels]))
      inputs = {'input_ids': params[0], 'attention_mask': params[1],
                LABEL_FIELD_DICT[self.model_name]: params[2]}
      outputs = self.model(**inputs)
      loss, logits = outputs[:2]
      log_posteriors = torch.log(torch.nn.Softmax(dim=2)(logits))
    results = []
    total_lp = 0.0
    for i, text in enumerate(texts):
      ids = tensor_input[i, :]
      lp = log_posteriors[i, :, :]
      sum_lp = 0.0
      for j, k in enumerate(ids.tolist()[1:]):
        if j + 1 >= lens[i]: break
        sum_lp += lp[j, k]
      results.append((text, sum_lp))
      total_lp += sum_lp
    total_lp_alternative = -loss * torch.sum(lens - 1)
    assert(torch.isclose(total_lp_alternative, total_lp)), \
        "{:.3f} ≠ {:.3f}".format(total_lp_alternative, total_lp)
    return results


def get_available_devices():
  return ['cpu'] + ['cuda:{}'.format(idx)
                    for idx in range(torch.cuda.device_count())]


if __name__ == "__main__":
  model_choices = list(LABEL_FIELD_DICT.keys())

  parser = argparse.ArgumentParser('Runninig LM on sample text from CLI')

  parser.add_argument('-m', '--model', help='model type', default='gpt2',
                      choices=model_choices)
  parser.add_argument('-d', '--device', help='device', default='cpu',
                      choices=get_available_devices())
  args = parser.parse_args()
  lm = LM(model_name=args.model, device=args.device)
  test_inputs = ["Hello, my dog is cute", "Hello, your dog is cute",
                 "Nothing is what everything isn't"]

  results = lm.score(test_inputs)
  for text, score in results:
      print(f"{score.item():.2f} {text}")

```

What was the issue?<|||||>MiniLM is not distilled with Masked LM task, only [Self-Attention distillation](https://github.com/huggingface/transformers/tree/master/model_cards/microsoft/MiniLM-L12-H384-uncased). It doesn't have LM head in the weights file. They are initialised randomly at each run 🤗 

```
{'missing_keys': ['cls.predictions.transform.dense.weight', 'cls.predictions.bias', 'cls.predictions.decoder.bias', 'cls.predictions.decoder.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias'], 'unexpected_keys': [], 'error_msgs': []}
```
