Hi @mrm8488.
Second your point, I am also thinking for storage space’s sake, we can host different preprocessing script. So when you say load dataset(Electra), it actually do load_dataset(wikipedia) and  load_dataset(bookcorpus) and go through electra preprocessing script, and for sure, cache it.
Before that come into reality, You definitely would want to check my work that pretrain Electra from the scratch, and fine tune on GLUE ,with fastai and huggin face/nlp.
repository: Pretrain-MLM-and-finetune-on-GLUE-with-fastai 18
the latest post of a series: [HuggingFace/nlp] Create fastai Dataloaders, show batch, and create dataset for LM, MLM 14
I have successfully finetune using hf model and reproduce the Electra small++ result, I am pretraining model from the scratch. You can get updates from my twitter Richard Wang 4