Awesome, great that you guys are already so interested in this 
The nice thing about OSCAR is that it is supported in the datasets library and we now also support datasets streaming: Load a Dataset in Streaming mode — datasets 1.8.0 documentation 6 which means that you surely won’t run into any disk space memory problems. We sadly don’t support this out of the box for mC4 (it’s not even in datasets yet), but I’d definitely recommend to try it out! Maybe you can even first train a model on OSCAR and then a second one on mC4