That’s really cool! mC4 can be painful depending on what languages you are dealing with. Streaming will be great for this.
Would it be viable to pre-train ALBERT or DeBERTa-v2 models in this event? Hard to make a decision on which one!