I was thinking of something similar to this. AFAIK, there are two problems in Colab. First, you have just two cores, which make the tokenization of very big datasets a bit slow. Let’s say you do so and save it to disk… then you have to face the second issue, IO performance of disks in colab-- check this issue Slow dataloading with big datasets issue persists · Issue #2252 · huggingface/datasets · GitHub 3
Performing something more modest –finetuning a BERT model on a dataset of some gigabytes– is still problematic because of this.