Fine tune Xlnet
Can anybody guide me on how to fine tune xlnet for simple text classification task or any reference code because i am lost.
As far as I know, the pytorch code of XLNet is not completely ready now. But you could find it in the branch `xlnet` and the classifier code is nearly ready in the file `example/run_xlnet_classifier.py`. I have successfully fine-tuned it on the SST-2 task (which belongs to GLUE) with following args:

```shell
python run_xlnet_classifier.py \
--data_dir ..\glue_data\SST-2 \
--task_name sst-2 \
--output_dir sst_model \
--do_train \
--do_eval \
--max_seq_length 128 \
--train_batch_size 64 \
--learning_rate 5e-6
```<|||||>@SivilTaram Cant be fine tuned on external data ? is the tensorflow version ready ?<|||||>@AhmedBahaaElDinMohammed Sure you could fine-tune it on external data, which means you should process your data and construct train/validate `examples` as SST-2 does. You could see `example/utlis_glue.py` for more details to handle your external data :)

The tensorflow is ready, you could refer to the original repo for help. This repo is only for pytorch version, thanks.<|||||>> I have successfully fine-tuned it on the SST-2 task (which belongs to GLUE) 

@SivilTaram Would it be possible to fine-tune it on SQuAD 2.0? or alternatively, convert a fine-tuned model from the original repo/tensorflow?<|||||>@edanweis Not ready now. Please wait the author to complete the awesome work :) Or you could watch the updates of PR [here](https://github.com/huggingface/pytorch-pretrained-BERT/pull/711).<|||||>Has anyone tried fp16 for xlnet?
I tried it and found that the memory was half, but it was slower than fp32(even when I used the same GPU memory). 
Environment: v100, cuda 10.0, torch 1.1
The environment is ok, because I tried bert + fp16 and it was much faster than fp32.
I thought it is the problem of torch.einsum, but I am not that sure. 
Guys, do you have the same problem ?<|||||>@SivilTaram Following the latest release 0.6.2, I am trying to convert my tf checkpoints:

```
export TRANSFO_XL_CHECKPOINT_PATH=home/edanweis/xlnet/model/squad
export TRANSFO_XL_CONFIG_PATH=home/edanweis/xlnet/model/squad
export FINETUNING_TASK=squad


pytorch_transformers xlnet \
  $TRANSFO_XL_CHECKPOINT_PATH \
  $TRANSFO_XL_CONFIG_PATH \
  $PYTORCH_DUMP_OUTPUT \
  $FINETUNING_TASK \
```
But getting `pytorch_transformers/__main__.py", line 111, in main FINETUNING_TASK) UnboundLocalError: local variable 'FINETUNING_TASK' referenced before assignment`<|||||>@SivilTaram Did you try to finetune XLNet with the last code (release 1.0) using examples/run_glue.py? Everything works but accuracy didn't change and every time is around 0.50? It looks like it didn't train at all. I used the following script:

```
export GLUE_DIR=/path/to/glue

python ./examples/run_glue.py \
    --model_type xlnet \
    --model_name_or_path xlnet-large-cased \
    --do_train  \
    --do_eval   \
    --evaluate_during_training \
    --logging_steps 500 \
    --save_steps 1000 \
    --task_name=sst-2     \
    --data_dir=${GLUE_DIR}/SST-2  \
    --output_dir=./proc_data/sst-2   \
    --max_seq_length=128   \
    --per_gpu_eval_batch_size=8   \
    --per_gpu_train_batch_size=8   \
    --gradient_accumulation_steps=1 \
    --max_steps=8000  \
    --model_name=xlnet-large-cased   \
    --overwrite_output_dir   \
    --overwrite_cache \
    --warmup_steps=120
```<|||||>@avostryakov I do not yet. I guess you could explore if the loss decrease as expected? There should be loss logs, along with tensorboard logs.<|||||>@SivilTaram Evaluation loss isn't changed, training loss is increased. It looks like something wrong with the optimization process during training.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
