How to save tokenize data when training from scratch
# ❓ Questions & Help

I am training Allbert from scratch following the blog post by hugging face. As it mentions that :

> If your dataset is very large, you can opt to load and tokenize examples on the fly, rather than as a preprocessing step.

How this can be done any suggestions?
As of now , using the method given in the notebook:
```
from transformers import TextDataset

dataset = TextDataset(
    tokenizer=tokenizer,
    file_path="./oscar.eo.txt",
    block_size=128,
)
```
 there is no method to save tokenize data, can anyone suggest how to save that as its already taking long enough before starting training. 

     



There is a method to save tokenizer. Check this notebook: https://github.com/huggingface/blog/blob/master/notebooks/01_how_to_train.ipynb

<|||||>Thats what I am using.
its saving it in the `dataset` variable not in any file. ByTokenize data I mean pretraining data.<|||||>You can look at serialization practices, you should able to do with torch at least.
https://huggingface.co/transformers/serialization.html#serialization-best-practices<|||||>well thats for all required model files. I am not getting how to save pretraining data<|||||>Once you have your data you can pickle it or use `torch.save` to save it to your disk and reload it later.<|||||>That worked @LysandreJik thanks! 
I still not getting how you can prepare pretraining data on the fly while training. I got large training data and don't want to wait until it gets prepared for training.<|||||>Have you taken a look at PyTorch's Dataset/Dataloader utilities? I recommend taking a look at [loading hude data functionality](https://discuss.pytorch.org/t/loading-huge-data-functionality/346) or [how to use a dataset larger than memory](https://discuss.pytorch.org/t/how-to-use-dataset-larger-than-memory/37785/8).

I personnally prefer using IterableDatasets when loading large files, as I find the API easier to use to limit large memory usage. This [tutorial](https://medium.com/swlh/how-to-use-pytorch-dataloaders-to-work-with-enormously-large-text-files-bbd672e955a0) is interesting on that subject.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
