Appropriate dataset format for language modeling example
# What is the most memory efficient way/best way to format your dataset file for language modeling?
## Details
I am running run_clm.py and can only get my dataset to work with the smallest GPT2 model. I would like to experiment with gpt2-xl ideally but would settle for large and xlnet. I am using distributed training on TPU assuming that this improves memory. I have saved my file in a .txt. It is roughly 20mb with 82,000 samples of mean length 256 std 250 and is line delimited (where each line signifies a sample).

Is this the correct approach? I notice the .raw files used in training, are these smaller? Is there a way to pretokenize?
I hesitate to ask this on stack as it is not a bug.
@sgugger sorry to bother if this is a common issue I'd love to hear more no worries if it will require much work I can focus on that. Just lost for resources.

Thank you all!
General questions should be asked on the [forum](https://discuss.huggingface.co/) as we keep the issues for bugs.
`run_clm` doesn't use the approach of taking the different lines of a dataset (that's in `run_mlm` as it's usually done for masked language modeling) so you will have to tweak the example script to do this.

There is no way to pretokenzie, but the result of the tokenization will be cached, so it will only be run once on a given machine.<|||||>Thank you for the help! I won't make that mistake again my apologies. Editing the clm file now. New to open source software and excited to dig in and help please bear with me while I learn the ropes!<|||||>No worries, I'm just telling you for next time :-)
Good luck with your scripting, closing this issue for now.