@thomwolf @lhoestq I want to join and work on adding larger datasets used for model pre-training. I’d start with preparing datasets used to train Alexa Bort. We already have Wikipedia, Bookcorpus, OpenWebText. I want to add Wiktionary, UrbanDictionary, One Billion Words, the news subset of Common Crawl (Nagel, 2016).
I’d like to contribute to the creation of datasets tooling so any researchers working on a nextgen LM can quickly and easily make arbitrary “brews” of these large datasets and use them in pre-training.
Cheers,
Vladimir