Can't load too big text file for dataset (RAM exhausts), HELP PLEASE !!!!!
Hello All,

I am trying to load a big text file for pertaining a BERT model from scratch the size of the txt file is about 11GB and trying to load the model for pertaining is exhausting all the RAM on the system.

Is it possible to load all the data in batches and then perform training.

I am a bit new to hugging face ecosystem so I would request you all to help me around if you have any clue about this.
I am using Google Colab for the purpose.

Please share code snippets, If possible.

Cheers !

`#construct dataset 
from transformers import LineByLineTextDataset

file_path="/content/drive/MyDrive/full_text_data.txt"
dataset = LineByLineTextDataset(
    tokenizer=tokenizer,
  file_path=file_path,
    block_size=32)`


(duplicate of #19199)