I have been in a dilemma for training language models. I wanted to train a model with large amount (>5TB) of data. What I did is,

Write my own data loading script using this 2.
Binarize the data
Load the data using load_dataset module
Use huggingface trainer to train the model. At first, I wrote my own training loop. But since hf-trainer comes with deepspeed I decided to move into it.
But what happens that my binarize data becomes large and it cannot load the full data into the RAM (not even in 428GB RAM).

I was wondering is there any way I can shard the data and perform lazy loading? What I mean by Lazy-loading is to load a binarized shard into memory only when I train the model with a specific shard.
The tradeoff becomes, either I learn deepspeed and integrate into my training loop which seems a very hectic job, or try solving this issue with hf-dataset.
@patrickvonplaten @valhalla