Can't load too big text file for dataset (RAM exhausts), HELP PLEASE !!!!!
### System Info

MacBook Pro (13-inch, M1, 2020)
Chip Apple M1
Memory 16 GB

(Also tried with Google Colab)


### Who can help?

copy of #19161

Hello All,

A bit new to HuggingFace environment.

I am trying to load a big text file for pertaining a BERT model from scratch the size of the txt file is about 11GB and trying to load the model for pertaining is exhausting all the RAM on the system.

Is it possible to load all the data in batches and then perform training.

I am a bit new to hugging face ecosystem so I would request you all to help me around if you have any clue about this.
I am using Google Colab for the purpose.

Please share code snippets, If possible.

Cheers !

`#construct dataset
from transformers import LineByLineTextDataset

file_path="/content/drive/MyDrive/full_text_data.txt"
dataset = LineByLineTextDataset(
tokenizer=tokenizer,
file_path=file_path,
block_size=32)`

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

try to load the data with a big text file 

### Expected behavior

The data should be read in the batches to combat OOM.
try using iterable dataset and decrease the batch_size<|||||>@Kunlun-Zhu 

I am a bit new to HF environment, can you please provide an example code snippet to do so ?

Thanks !!<|||||>> @Kunlun-Zhu
> 
> I am a bit new to HF environment, can you please provide an example code snippet to do so ?
> 
> Thanks !!

https://pytorch.org/docs/stable/data.html
more examples could be found on stackoverflow and many blogs<|||||>Hi @mv96 👋 

As per our [issues guidelines](https://github.com/huggingface/transformers/blob/main/ISSUES.md), we reserve GitHub issues for bugs in the repository and/or feature requests. For any other matters, we'd like to invite you to use our [forum](https://discuss.huggingface.co/) 🤗<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.