How to fine tune Transformer-XL on own dataset?
I have my own dataset. What format of data I need to fine tune TransformerXL for text generation?
Hi,
You can tokenize your data with the Transformer-XL tokenizer and use it to train your model.<|||||>I understand it. But I want to fine tune pretrained model<|||||>What is the difference?

Can you start by trying to do what you would do normally to fine-tune a pytorch model and then if you encounter an issue give the error messages back here so we can help?<|||||>I found example in readme
And my question - can I fine tune model using this script with my own data? 
` python run_openai_gpt.py \
  --model_name openai-gpt \
  --do_train \
  --do_eval \
  --train_dataset $ROC_STORIES_DIR/cloze_test_val__spring2016\ -\ cloze_test_ALL_val.csv \
  --eval_dataset $ROC_STORIES_DIR/cloze_test_test__spring2016\ -\ cloze_test_ALL_test.csv \
  --output_dir ../log \
  --train_batch_size 16 \ `

Example for gpt, but I guess it is the same for TransformerXL<|||||>Hi, you will likely need to adapt this example since Transformer-XL uses memory cells but there is no ready to use example for fine-tuning Transformer-XL in the repo unfortunately (and I don't plan to add one in the near future).

If you want to give it a try feel free to ask more specific questions here.

I would advise to start by reading carefully the author's paper to be sure you have a good understanding of the model. You can also probably start from the authors' own PyTorch training script (which I simplified to make the evaluation script in the present repo).<|||||>Is it possible to use the transformer with non-tokenized data @thomwolf? I know the Transformer is built for language modeling, but I would like to take advantage of the self-attention aspect of the Transformer's self-attention mechanism to model continuous data, and wonder how doable that sounds to you? <|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
