Can't reproduce XNLI zero-shot results from MBERT in Chinese
## ❓ Questions & Help

Hi guys,
I am trying to reproduce the XNLI zero-shot transfer results from MBERT.
With the same code and same checkpoint but different language for the test set, I am not able to reproduce the results for Chinese, Arabic, and Urdu. Does anyone encounter the same problem? Thanks!


Model |English | Chinese | Arabic | German | Spanish | Urdu
-- |-- | -- | -- | -- | -- | --
From MBERT github page | 81.4 | 63.8 | 62.1 | 70.5 | 74.3 | 58.3
My results | 82.076 | 36.088 | 35.327 | 68.782 | 70.419 | 35.170




Hey @edchengg, I'm running into the same problem. Were you able to figure this out? thanks
<|||||>Anyone else who stumbles here.
**Fix**: Just use bert-based-multilingual-cased as shown here https://huggingface.co/transformers/v2.3.0/examples.html.
When I used Google's mBERT and made it pyTorch compatible using the convert_tf_original..... script from src/transformers, somehow it doesn't learn properly. Couldn't figure out why, hence opening a new issue here : https://github.com/huggingface/transformers/issues/5019<|||||>@bsinghpratap Did you manage to run the script in only evaluation mode? When I try to evaluate an mBERT model trained on MNLI, it just freezes at 99%.<|||||>Yeah, I ran it on eval mode as well. Works fine for me.