Error when Fine-tuning XLM on SQuA
## 🐛 Bug

<!-- Important information -->

Model I am using (XLM):

Language I am using the model on (English and Chinese):

The problem arise when using:

[CUDA_VISIBLE_DEVICES=2 python run_squad.py   --model_type xlm  --model_name_or_path xlm-mlm-tlm-xnli15-1024   --do_train   --do_eval     --train_file $SQUAD_DIR/Crosslingual_QZH_train.json   --predict_file $SQUAD_DIR/Crosslingual_QZH_valid.json   --per_gpu_train_batch_size 12   --learning_rate 3e-5   --num_train_epochs 100   --max_seq_length 384   --doc_stride 128 --eval_all_checkpoints --save_steps 50 --evaluate_during_training  --output_dir /home/weihua/Sqad/transformers/xlm_out/ ] 


error:
Traceback (most recent call last):
  File "run_squad.py", line 569, in <module>
    main()
  File "run_squad.py", line 515, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File "run_squad.py", line 179, in train
    results = evaluate(args, model, tokenizer)
  File "run_squad.py", line 275, in evaluate
    args.version_2_with_negative, tokenizer, args.verbose_logging)
  File "/home/weihua/Sqad/transformers/examples/utils_squad.py", line 814, in write_predictions_extended
    final_text = get_final_text(tok_text, orig_text, tokenizer.do_lower_case,
AttributeError: 'XLMTokenizer' object has no attribute 'do_lower_case'

![WeChat Image_20191108182243](https://user-images.githubusercontent.com/43492059/68469281-d0188e00-0254-11ea-928a-8ab4a7140838.png)
![WeChat Image_20191108182248](https://user-images.githubusercontent.com/43492059/68469288-d3ac1500-0254-11ea-9116-9f0dd15b65cd.png)


How can I deal with this problem?



> ##  Bug
> Model I am using (XLM):
> 
> Language I am using the model on (English and Chinese):
> 
> The problem arise when using:
> 
> [CUDA_VISIBLE_DEVICES=2 python run_squad.py --model_type xlm --model_name_or_path xlm-mlm-tlm-xnli15-1024 --do_train --do_eval --train_file $SQUAD_DIR/Crosslingual_QZH_train.json --predict_file $SQUAD_DIR/Crosslingual_QZH_valid.json --per_gpu_train_batch_size 12 --learning_rate 3e-5 --num_train_epochs 100 --max_seq_length 384 --doc_stride 128 --eval_all_checkpoints --save_steps 50 --evaluate_during_training --output_dir /home/weihua/Sqad/transformers/xlm_out/ ]
> 
> error:
> Traceback (most recent call last):
> File "run_squad.py", line 569, in 
> main()
> File "run_squad.py", line 515, in main
> global_step, tr_loss = train(args, train_dataset, model, tokenizer)
> File "run_squad.py", line 179, in train
> results = evaluate(args, model, tokenizer)
> File "run_squad.py", line 275, in evaluate
> args.version_2_with_negative, tokenizer, args.verbose_logging)
> File "/home/weihua/Sqad/transformers/examples/utils_squad.py", line 814, in write_predictions_extended
> final_text = get_final_text(tok_text, orig_text, tokenizer.do_lower_case,
> AttributeError: 'XLMTokenizer' object has no attribute 'do_lower_case'
> 
> ![WeChat Image_20191108182243](https://user-images.githubusercontent.com/43492059/68469281-d0188e00-0254-11ea-928a-8ab4a7140838.png)
> ![WeChat Image_20191108182248](https://user-images.githubusercontent.com/43492059/68469288-d3ac1500-0254-11ea-9116-9f0dd15b65cd.png)
> 
> How can I deal with this problem?

Look [here](https://github.com/huggingface/transformers/blob/master/transformers/tokenization_xlm.py). This is the source code of the XLM tokenizer. In the ___init__()_ method, you can see that the _do_lower_case_ parameter doesn't exist, but it exists the **do_lowercase_and_remove_accent** parameter.

Said this, I tell you a solution to your problem (a wordaround).
If you see the source code of [run_squad.py](https://github.com/huggingface/transformers/blob/master/examples/run_squad.py) at rows 487-489, you see that here the tokenizer (different according to the model chosen by you - in your case XLM) is loaded, passing the _do_lower_case_ parameter. But as I said before, the tokenizer for XLM model accepts a different parameter. So, you can insert an if/else statement (in pseudo-code):
```
if args.model_type == 'XLM':
    tokenizer = tokenizer_class.from_pretrained(..., do_lower_case_and_remove_accent=args.do_lower_case, ...)
else:
    tokenizer = tokenizer_class.from_pretrained(..., do_lower_case=args.do_lower_case, ...)
```
N.B. if you want to be more comfortable of my solution to your problem, please see the source code of the `tokenization_xlm.py` in your HuggingFace version whether it contains the _do_lowercase_and_remove_accent_ parameter and not the _do_lowercase_ parameter. 

Keep us updated on your problem! Good luck<|||||>Actually, I just replace the line 814 in utils_squad.py with 'final_text = get_final_text(tok_text, orig_text,verbose_logging)'. It means I no longer pass ‘tokenizer.do_lower_case’ to this function. Because I find that in the tokenization_xlm.py, the parameter 'do_lower_case_and_remove_accent' is already set. Although this allows my program to run, it does not give good results. So I suspect that my change is wrong.
I also tried to follow your suggestion above. But the results did not improve. 
This is the result of the 700th step. 
![WeChat Image_20191108204610](https://user-images.githubusercontent.com/43492059/68477603-d6b10080-0268-11ea-947f-995b157da8d3.png)
The exact and f1 is very low. I used bert-base-multilingual-cased to do SQuAD with the same data set before. The exact and f1 increased very fast. At around 200th step, the f1 can reach 40+, and exact is about 30.
Is it wrong with my parameter settings that caused these two scores to rise slowly when I use XLM?
Or is it necessary to take longer to finetune the SQuAD task in XLM than in BERT?
This is my first time to do the SQuAD. I sincerely hope that you can give me some advice.
Thank you.<|||||>I also tried to train and test on the official dataset,
that is 
![WeChat Image_20191109175459](https://user-images.githubusercontent.com/43492059/68526753-13890000-031a-11ea-9f38-e9841bb0ac42.png)
in the examples document.

but the situation was the same as what I got on my own dataset.
At the same time, I find that when I use the XLM model with learning-rate= 3e-5, the loss decreases very slowly. 
Is this due to the inappropriateness of my parameter settings? Or is it because the loss function of the model needs to be changed?<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
