Attention heads attend equally after conversion from tensorflow checkpoint
# 🐛 Bug

## Information

Hi.

I'm using notebook https://github.com/jessevig/bertviz/blob/master/head_view_bert.ipynb from https://github.com/jessevig/bertviz for visualizing attention in the Bert model. Given example works fine with pytorch default models. The problem arises when I'm converting pre-trained (with https://github.com/google-research/bert#pre-training-with-bert) on custom dataset Bert-Base Multilingual Uncased model.

Pytorch default model visualization:
![Screenshot from 2020-06-30 12-08-01](https://user-images.githubusercontent.com/22409396/87297669-7bc75b80-c511-11ea-8ca1-a24f8d4c31df.png)

Example of attention values for default model:

         [[8.3652e-03, 6.9530e-02, 6.6828e-02,  ..., 9.5115e-03, 2.7546e-02, 4.5171e-01],
          [3.3669e-03, 1.2537e-02, 9.2709e-03,  ..., 1.5638e-03, 1.0154e-03, 9.1897e-01],
          [6.5795e-03, 3.6612e-03, 5.4454e-02,  ..., 1.1923e-03, 4.1071e-03, 8.4899e-01],
          ...,
          [7.3705e-03, 2.5430e-03, 7.6645e-03,  ..., 1.7184e-02, 4.6256e-02, 8.2301e-01],
          [2.2311e-02, 1.8006e-03, 4.3833e-02,  ..., 9.1167e-03, 1.3746e-01, 6.2295e-01],
          [7.5967e-02, 4.2936e-02, 4.6500e-02,  ..., 4.9925e-02, 6.6538e-02, 5.0721e-02]],

         [[3.5124e-02, 2.2295e-02, 9.2680e-03,  ..., 1.1409e-02, 1.7234e-02, 5.5768e-01],
          [7.0571e-03, 3.7321e-01, 1.7890e-02,  ..., 7.6114e-03, 8.8965e-03, 3.6259e-01],
          [8.8010e-03, 4.9023e-03, 1.4315e-01,  ..., 2.2279e-03, 7.9276e-02, 4.3233e-01],
          ...,

Visualization after model conversion from tf to pytorch:
![Screenshot from 2020-06-30 12-07-42](https://user-images.githubusercontent.com/22409396/87297740-97326680-c511-11ea-8a18-6a33160034ab.png)

Example of attention values after conversion:

          [[0.0716, 0.0686, 0.0556,  ..., 0.0776, 0.0783, 0.0648],
          [0.0893, 0.0513, 0.0641,  ..., 0.0606, 0.0908, 0.0554],
          [0.0868, 0.0663, 0.0621,  ..., 0.0822, 0.0777, 0.0471],
          ...,
          [0.0906, 0.0750, 0.0649,  ..., 0.0807, 0.1011, 0.0444],
          [0.0670, 0.0667, 0.0620,  ..., 0.0877, 0.0739, 0.0515],
          [0.0773, 0.0738, 0.0652,  ..., 0.0787, 0.0856, 0.0518]],

         [[0.0553, 0.0622, 0.0665,  ..., 0.0585, 0.0845, 0.0670],
          [0.0631, 0.0829, 0.0592,  ..., 0.0608, 0.0968, 0.0532],
          [0.0561, 0.0720, 0.0617,  ..., 0.0628, 0.1010, 0.0802],
          ...,

Another experiments:
1. loading tensorflow checkpoint directly without conversion - works fine, not equal attentions;
2. loading pytorch model after saving it from loaded tensorflow checkpoint also works fine;
3. tested with Bert-Base Multilingual Uncased model without pre-training (to be sure that pre-training doesn't cause the problem) - got the same results.

So, I guess that conversion from tf checkpoint works wrong or I'm converting model in wrong way.
Any explanation of described behavior would be appreciated, thank you.

## To reproduce

Steps to reproduce the behavior:

1. Example from https://github.com/jessevig/bertviz/blob/master/head_view_bert.ipynb - works fine.
2. Conversion tensorflow checkpoint to pytorch gives incorrect attentions - not work. 

```python
from transformers import convert_bert_original_tf_checkpoint_to_pytorch

convert_bert_original_tf_checkpoint_to_pytorch.convert_tf_checkpoint_to_pytorch(
    'model/multilingual_L-12_H-768_A-12/bert_model.ckpt.index', 
    'model/multilingual_L-12_H-768_A-12/bert_config.json', 
    'model/multilingual_L-12_H-768_A-12/pytorch_model.bin')
```
Then I copy converted pytorch_model.bin, config.json, vocab.txt to model/multilingual_L-12_H-768_A-12_pytorch/

```python
from transformers import BertTokenizer,  BertForPreTraining

pytorch_bert_model = 'model/multilingual_L-12_H-768_A-12_pytorch/'

model = BertForPreTraining.from_pretrained(pytorch_bert_model)
tokenizer = BertTokenizer.from_pretrained(pytorch_bert_model, do_lower_case=True)
```
No errors, but attention values seems wrong as written above. 
```
INFO:transformers.configuration_utils:loading configuration file model/multilingual_L-12_H-768_A-12_pytorch/config.json
INFO:transformers.configuration_utils:Model config BertConfig {
  "attention_probs_dropout_prob": 0.1,
  "directionality": "bidi",
  "gradient_checkpointing": false,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 512,
  "model_type": "bert",
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "output_attentions": true,
  "pad_token_id": 0,
  "pooler_fc_size": 768,
  "pooler_num_attention_heads": 12,
  "pooler_num_fc_layers": 3,
  "pooler_size_per_head": 128,
  "pooler_type": "first_token_transform",
  "type_vocab_size": 2,
  "vocab_size": 105879
}

INFO:transformers.modeling_utils:loading weights file model/multilingual_L-12_H-768_A-12_pytorch/pytorch_model.bin
INFO:transformers.modeling_utils:All model checkpoint weights were used when initializing BertForPreTraining.

INFO:transformers.modeling_utils:All the weights of BertForPreTraining were initialized from the model checkpoint at model/multilingual_L-12_H-768_A-12_pytorch/.
If your task is similar to the task the model of the ckeckpoint was trained on, you can already use BertForPreTraining for predictions without further training.

INFO:transformers.tokenization_utils_base:loading file model/multilingual_L-12_H-768_A-12_pytorch/vocab.txt
```
Using ```BertModel``` for loading converted model gives the same result.

3. Loading tensorflow checkpoint directly without conversion - works fine

```python
from transformers import BertTokenizer,  BertForPreTraining

bert_config_file = 'model/multilingual_L-12_H-768_A-12_pytorch/config.json'
bert_vocab_file = 'model/multilingual_L-12_H-768_A-12_pytorch/vocab.txt'
tf_bert_checkpoint ='model/multilingual_L-12_H-768_A-12/bert_model.ckpt.index'

model = BertForPreTraining.from_pretrained(tf_bert_checkpoint, from_tf=True, config=bert_config_file)
tokenizer = BertTokenizer.from_pretrained(bert_vocab_file, do_lower_case=True)
```

4. Loading pytorch model after saving it from loaded tensorflow checkpoint - works fine

```python
from transformers import BertTokenizer,  BertForPreTraining

bert_config_file = 'model/multilingual_L-12_H-768_A-12_pytorch/config.json'
bert_vocab_file = 'model/multilingual_L-12_H-768_A-12_pytorch/vocab.txt'
tf_bert_checkpoint ='model/multilingual_L-12_H-768_A-12/bert_model.ckpt.index'

model = BertForPreTraining.from_pretrained(tf_bert_checkpoint, from_tf=True, config=bert_config_file)

model.save_pretrained(model/test/')

pytorch_bert_checkpoint_path = 'model/test/pytorch_model.bin'
model = BertForPreTraining.from_pretrained(pytorch_bert_checkpoint_path, config=bert_config_file)
tokenizer = BertTokenizer.from_pretrained(bert_vocab_file, do_lower_case=True)
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

Conversion from tensorflow checkpoint do not influence on model attentions (values doesn't seem equal), attentions can be visualized correctly.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.0.2
- Platform: Ubuntu 18.04.3 LTS
- Python version: 3.7.7
- PyTorch version (GPU): 1.4.0
- Tensorflow version (GPU): 1.15.0
- Using GPU in script: No
- Using distributed or parallel set-up in script: No

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
