@Batik so the easiest way to overcome all the RAM limitation will likely be to use the new nlp library which is specifically developed for that.
Jared, a community contributor, is actually adding right now a loader for text dataset which is probably exactly what you’ll need. You can follow the PR here: https://github.com/huggingface/nlp/pull/356 222
I’ll ping him as well (he’s not yet on the forum I think).