Not able to reproduce XNLI results from Google's mBERT weights.
# 🐛 Bug

Hello, 
I am using mBERT from https://github.com/google-research/bert/blob/master/multilingual.md and using it with the run_xnli.py file.

Language: Learning the model on English (en) and testing it on French (fr).

The problem arises when I download the mBERT's weights from: https://github.com/google-research/bert/blob/master/multilingual.md and then use https://github.com/huggingface/transformers/blob/master/src/transformers/convert_bert_original_tf_checkpoint_to_pytorch.py to get the PyTorch compatible weights.

I then used the PyTorch compatible weights with run_xnli provided here: https://github.com/huggingface/transformers/blob/master/examples/text-classification/run_xnli.py. Using the language as fr and train_language as en. The model doesn't learn properly and doesn't provide performance above 0.40 on fr. 
Though if I use the weights directly provided by huggingface for bert-base-multilingual-cased (as mentioned here: https://huggingface.co/transformers/v2.3.0/examples.html). It works fine and the mBERT model is able to achieve SOTA performance. Would it be possible to understand as why this might be happening?

The tasks I am working on is XNLI. Learning on English (en) and zero-shot testing on French (fr).

Steps to reproduce the behavior:
1. Get the mBERT model from the link provided here: https://github.com/google-research/bert/blob/master/multilingual.md
2.Get the PyTorch compatible weights using: https://github.com/huggingface/transformers/blob/master/src/transformers/convert_bert_original_tf_checkpoint_to_pytorch.py
3. Copy the vocab.txt and config.json from TF weights. Add "model_type"="bert" to config.
4. run the example run_xnli.py provided here: https://github.com/huggingface/transformers/blob/master/examples/text-classification/run_xnli.py
5. Default parameters can be used with model_path set to the PyTorch weights of mBERT.
     
- `transformers` version: 2.11
- Python version: 3.7
- PyTorch version (GPU): 1.5.0
- Tensorflow version (GPU): 2.2.0
- Using GPU in script: Yes
- Using parallel setup across 3 1080ti GPUs

Edit: For some reason the model: xlm-mlm-tlm-xnli15-1024 (https://huggingface.co/transformers/pretrained_models.html), also doesn't learn properly. The train accuracy never goes beyond 0.35.

Fix for XLM is provided here: https://github.com/huggingface/transformers/pull/5035

The issue of mBERT (weights from Google) still not resolved.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
