Fine Tuning GPT2 on wikitext-103-raw
## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Running pytorch-transformers\examples\run_lm_finetuning.py. This is stuck at  load_and_cache_examples step. I just see message like
  WARNING - pytorch_transformers.tokenization_utils -   This tokenizer does not make use of special tokens. The sequence has been returned with no modification.
The train file has 1.8M rows with this rate it would take few days to just tokenize and cache training data.
Is this expected?  Did anyone face this before?

Thanks in advance for your help.
@snaik2016 I ran into the same issue and had to parallelize my code to make it faster. Also getting rid of the while loop and list splicing in the TextDataset class with a for loop made it much quicker.<|||||>Please check #1830 . I made some tuning on a training part. But I guess it'll still take many days for 1.8M rows dataset (in fact, talking about the token count rather than rows count is more meaningful) .<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
