XLNet large low accuracy
I was running run_glue.py on one of my classification problems. Using XLNet-base-cased, everything seems to be fine, the classification accuracy converge to around 92%. But using XLNet-large, the accuracy is 89% at the first checkpoint and then drop into 24.85% at the second checkpoint. The data should be ok because I have been running many different algorithms on it. 

Some of the logs as below. What could be the possible cause?

```
07/31/2019 05:35:28 - INFO - __main__ -   Saving features into cached file /data/_working/sentiment/large/cached_dev_xlnet-large-cased_128_sentimentall
07/31/2019 05:35:29 - INFO - __main__ -   ***** Running evaluation  *****
07/31/2019 05:35:29 - INFO - __main__ -     Num examples = 10000
07/31/2019 05:35:29 - INFO - __main__ -     Batch size = 8
07/31/2019 05:38:49 - INFO - __main__ -   ***** Eval results  *****
07/31/2019 05:38:49 - INFO - __main__ -     acc = 0.8902
07/31/2019 05:38:55 - INFO - __main__ -   Saving model checkpoint to /data/_working/sentiment/large/output/checkpoint-500000/1250 [03:19<00:00,  6.16it/s]
07/31/2019 13:56:16 - INFO - __main__ -   Loading features from cached file /data/_working/sentiment/large/cached_dev_xlnet-large-cased_128_sentimentalls]
07/31/2019 13:56:16 - INFO - __main__ -   ***** Running evaluation  *****
07/31/2019 13:56:16 - INFO - __main__ -     Num examples = 10000
07/31/2019 13:56:16 - INFO - __main__ -     Batch size = 8
07/31/2019 13:59:39 - INFO - __main__ -   ***** Eval results  *****
07/31/2019 13:59:39 - INFO - __main__ -     acc = 0.2485
07/31/2019 13:59:44 - INFO - __main__ -   Saving model checkpoint to /data/_working/sentiment/large/output/checkpoint-100000/1250 [03:22<00:00,  6.16it/s]
 Iteration:  72%|██████████████████████████████████████████████████████████████████▉                          | 142278/197500 [23:46:00<9:18:52,  1.65it/s]07/31/2019 22:22:27 - INFO - __main__ -   Loading features from cached file /data/_working/sentiment/large/cached_dev_xlnet-large-cased_128_sentimentalls]
07/31/2019 22:22:27 - INFO - __main__ -   ***** Running evaluation  *****
07/31/2019 22:22:27 - INFO - __main__ -     Num examples = 10000
07/31/2019 22:22:27 - INFO - __main__ -     Batch size = 8
07/31/2019 22:25:49 - INFO - __main__ -   ***** Eval results  *****
07/31/2019 22:25:49 - INFO - __main__ -     acc = 0.2485
07/31/2019 22:25:55 - INFO - __main__ -   Saving model checkpoint to /data/_working/sentiment/large/output/checkpoint-150000/1250 [03:21<00:00,  6.00it/s]

```
The first thought could be that the learning rate is too high and you overfit.
You probably should try changing the batch size too.
You can have a look at #795 where we discussed similar questions for SST-2. <|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
