LM Fine-tuning for XLNET?
## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

Hi there, is there any plan to add support for fine-tuning XLNET? This one currently isn't available in run_lm_finetuning.py.

Thanks!
I'm also looking forward to it! <|||||>I might be naive here, but will adding xlnet's config, LMHeadModel, and tokenizer to MODEL_CLASSES work?
https://github.com/huggingface/transformers/blob/74ce8de7d8e0375a9123f9542f3483f46cc8df9b/examples/run_lm_finetuning.py#L61<|||||>I don't think adding the Python classes for XLNet is enough. The code distinguishes between two modes of fine-tuning a language model: 1. masked language models (MLM), i.e. BERT and company 2. Sequential left-to-right LM, GPT and GPT2. 

Check this, for example:
https://github.com/huggingface/transformers/blob/74ce8de7d8e0375a9123f9542f3483f46cc8df9b/examples/run_lm_finetuning.py#L224

XLNet is neither. Its LM task is to predict factorization permutations (check paper for more details) of a given sequence. In simple words, it splits in a sequence into two sequential sub-sequences in all possible ways and it uses the first one as input for predicting the second one.

The original implementation has sample code on how to produce the factorization permutation:
https://github.com/zihangdai/xlnet/blob/5cd50bc451436e188a8e7fea15358d5a8c916b72/data_utils.py#L579<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>Pretty much looking forward to it