Tokenizing/Loading Data for GPT-2 (1 example per line)
I want to finetune GPT-2 on a dataset that has one example per line, and the examples all have different length. What is the best way to alter the TextDataset class to allow for this? I understand GPT-2 requires fixed length inputs but I'm not sure how to apply the attention mask to achieve this? Also do I need to add a bos/eos token to each line?

Appreciate the help
Please check this out: https://github.com/huggingface/transformers/issues/1816#issuecomment-554759751
This finetuning script https://github.com/huggingface/transformers/blob/master/examples/run_lm_finetuning.py and the library do almost everything for you. Thus you don't need to split the text into fixed blocksize etc. Just give it one datafile for training and one for evaluating.<|||||>I was using the script but when generating samples it seemed like it was mixing information from multiple examples in the training set, when they should be treated as individual examples. Is there a way to avoid this?<|||||>Like in this thread it mentions not concatenating different articles in the same input: https://github.com/huggingface/transformers/issues/1145#issuecomment-527225854? Is there a way to do that with single examples (if they are not correlated)<|||||>Language models look for entire corpus. You can seperate them via `<|endoftext|>` special token. The model learns (and already learned if you based it on a pretrained one while finetuning) to separate contexts by `<|endoftext|>` token. This is a common case for language models. For example, XLNet uses `<eod>` as separator. This is also implied at the original GPT paper while covering how to train GPT for document classification etc. See figure one: https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf
But, still if you want to make sure that different rows in the data are not even connected while training, then you can pad each line until fill `block_size`.  For example, GPT2 has 1024 input size and you have a data long 1540 (=1024+516) as tokens. Then you could have: 
<1024 tokens><516 tokens + 508 padding><nextdata..>
Thus you can make sure that GPT2 doesn't get mixed data as input.<|||||>Also, please check this tutorial http://jalammar.github.io/illustrated-gpt2/ by Jay Alammar<|||||>I thought GPT-2 didn't use padding tokens https://github.com/huggingface/transformers/issues/1435#issuecomment-538779523 and it's unclear to me how to use the attention mask. I'll try just using the <|endoftext|> token however<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
