Crosslingual classification with XLM, loss does not converge
## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I am trying to use the XLM pretrained model `xlm-mlm-tlm-xnli15-1024` for a cross lingual classification task, but I cannot get the loss to converge and the final accuracy is random. 

To check this was not an implementation error of my own doing, I ran the `run_xnli.py` example and found using `xlm-mlm-tlm-xnli15-1024` results in an accuracy of 30% while using `bert-base-multilingual-cased` results in the expected accuracy of 70%.

system config:
```
Platform Linux-4.4.0-1098-aws-x86_64-with-debian-stretch-sid
Python 3.6.9 |Anaconda, Inc.| (default, Jul 30 2019, 19:07:31) 
[GCC 7.3.0]
PyTorch 1.2.0+cu92
Tensorflow 2.0.0
```
I had the same problem with different tasks . I've tried all the XLM pre-training models and got random results. Please let us know if you have solved this problem. I'm trying to figure it out. @DanKing1903 <|||||>I was able to reproduce the results of XLM on XNLI.
It was highly sensitive to hyper parameters.
I would suggest that you tune your learning_rate ~ 1.5e-6 and batch_size ~ 4.<|||||>> I was able to reproduce the results of XLM on XNLI.
> It was highly sensitive to hyper parameters.
> I would suggest that you tune your learning_rate ~ 1.5e-6 and batch_size ~ 4.

Thank you for your answer. I modified the parameters as you suggested. xlm-mlm-17-1280 and xlm-mlm-100-1280 batch_size only go up to 2, and others modified the parameters as before. I wonder if it has anything to do with the task. Looking forward to your reply<|||||>I got the same problem.
I suggest you to go with RMSprop (require less memory than Adam so you can have a bigger batch size) with learning rate 3e-5 (very important to use a small learning rate otherwise it diverge) and clipnorm of 1.0. Personally, I use a global batch size of 20 where each GPU has a batch size of 10. I haven't tested with accumulated gradient since tf2.0 does not have a wrapper for it at the moment, but I think it will help.
It might also help adding momentum to RMSprop or a scheduling learning rate, but haven't test it yet. If you have some hint or previous experience on it please let me know
<|||||>> > I was able to reproduce the results of XLM on XNLI.
> > It was highly sensitive to hyper parameters.
> > I would suggest that you tune your learning_rate ~ 1.5e-6 and batch_size ~ 4.
> 
> Thank you for your answer. I modified the parameters as you suggested. xlm-mlm-17-1280 and xlm-mlm-100-1280 batch_size only go up to 2, and others modified the parameters as before. I wonder if it has anything to do with the task. Looking forward to your reply

In my limited experience, XLM is highly sensitive to HPs (it seems to be also the case with RoBERTa on GLUE, to a lesser extent). However, it is not something I observed with mBERT (and Distil-mBERT). So I don't think it has to do with XNLI since there is no consistent pattern across different models.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>Has anyone come up with a good set of hyper-parameters to train XLM models very well? Thanks for sharing the experience!