Indeed, we provide the Toronto Book Corpus 54 and Wikipedia 18 (i.e. the datasets used to pretrain Bert) but we are happy to welcome more datasets. Currently we are in the process of adding a very large multilingual dataset based on CommonCrawl and called “OSCAR”. You can follow this process here: https://github.com/huggingface/nlp/pull/348 15