Xlnet, Alberta, Roberta are not finetuned for CoLA task
## 🐛 Bug

I am currently trying to finetune pretrained models on CoLA task by using run_glue.py. Some of the models such as Bert and DistilBert are finetuned correctly as it is expected (the training loss goes down and the evaluation result is as what has been reported). Even though, for other models such as Roberta, Albert and Xlnet the training loss remains the same. I finetune the models for CoLA accordingly:

python run_glue.py --data_dir=./glue_data/CoLA/  --model_type=roberta 
--model_name_or_path=roberta-base --task_name=CoLA 
--output_dir=./model_roberta/  --max_seq_len=128 --do_train  --do_eval --num_train_epochs=3.0 --save_steps=50  --learning_rate=5e-5

## Observed behavior
(for Roberta-base)
"loss": 0.5889998215436936, "step": 50
"loss": 0.649243945479393, "step": 100
"loss": 0.6612952649593353, "step": 150
"loss": 0.6241107112169266, "step": 200
...
"loss":  0.6236384356021881, "step": 50
...
"loss": 0.6253059101104737, "step": 800
...
(I trained for more epochs (different lr )and still, the loss is near 0.5-0.6)
By debugging the code it seems that the model's output (softmax of logits) during training no matter what is the input is always label 1.

Another hint: I tried to finetune Roberta for other tasks such as STS-B and it finetuned well (loss was going down and the output of the model was not identical for all different inputs).

I was wondering if someone also has faced the same issue. How should I solve this?

OS type and version: Linux-3.10.0
Python: 3.7
Pytorch: 1.3.1

NM, I was able to solve it by changing some of the hyperparameter values.<|||||>Hi, glad you could make it work! Do you mind sharing what hyperparameter values you tuned in order to make it work?<|||||>> Hi, glad you could make it work! Do you mind sharing what hyperparameter values you tuned in order to make it work?

Hi, have you solved the problem? I used bert-base-cased, roberta-base, and xlnet-base-cased to predict CoLA test.tsv, and I got 51.8, 55.6 and 24.7 respectively, I don't know why xlnet got such low Matthew's Corr. Can you help me? thx 