[UPDATE] I tried subset of entire dataset.
66,101,891 lines, 5.1 G ( entire datasets ) : out of memory
15,000,000 lines: out of memory
3,000,000 lines, 231M: success
Can I use subset to train tokenizer and use it to train LM with entire datasets?