The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora
  with Web Data, and Web Data Only
Large language models are commonly trained on a mixture of filtered web data
and curated high-quality corpora, such as social media conversations, books, or
technical papers. This curation process is believed to be necessary to produce
performant models with broad zero-shot generalization abilities. However, as
larger models requiring pretraining on trillions of tokens are considered, it
is unclear how scalable is curation and whether we will run out of unique
high-quality data soon. At variance with previous beliefs, we show that
properly filtered and deduplicated web data alone can lead to powerful models;
even significantly outperforming models from the state-of-the-art trained on
The Pile. Despite extensive filtering, the high-quality data we extract from
the web is still plentiful, and we are able to obtain five trillion tokens from
CommonCrawl. We publicly release an extract of 600 billion tokens from our
RefinedWeb dataset, and 1.3/7.5B parameters language models trained on it.
647d5098396de7684dec2d0b
Guilherme Penedo
claimed_verified
2023-06-06T08:39:51.360Z
62596f9e1c0a084224b93e00
https://cdn-avatars.huggingface.co/v1/production/uploads/62596f9e1c0a084224b93e00/X2aLkJ0ofhkXwAg7lXvxD.jpeg
Guilherme Penedo
user
guipenedo
647d5098396de7684dec2d0c
Quentin Malartic
647d5098396de7684dec2d0d
Daniel Hesslow
admin_assigned
2023-06-05T16:38:54.213Z
61a8d1aac664736898ffc84f
https://cdn-avatars.huggingface.co/v1/production/uploads/1654022901910-61a8d1aac664736898ffc84f.jpeg
Daniel Hesslow
user
DanielHesslow
647d5098396de7684dec2d0e
Ruxandra Cojocaru
647d5098396de7684dec2d0f
Alessandro Cappelli
claimed_verified
2023-06-07T10:09:45.941Z
647edf57d26579210f58bcf3
https://cdn-avatars.huggingface.co/v1/production/uploads/647edf57d26579210f58bcf3/jyZbIiYxcS6s_i9FbDxJv.jpeg
alessandro cappelli
user
alecap
647d5098396de7684dec2d10
Hamza Alobeidli
647d5098396de7684dec2d11
Baptiste Pannier
647d5098396de7684dec2d12
Ebtesam Almazrouei
647d5098396de7684dec2d13
Julien Launay
claimed_verified
2023-06-06T08:39:54.455Z
60938bb4c65685640e32a7df
https://cdn-avatars.huggingface.co/v1/production/uploads/1620282175694-noauth.png
Julien Launay
user
slippylolo
The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora
  with Web Data, and Web Data Only
Large language models are commonly trained on a mixture of filtered web data
and curated high-quality corpora, such as social media conversations, books, or
technical papers. This curation process is believed to be necessary to produce
performant models with broad zero-shot generalization abilities. However, as
larger models requiring pretraining on trillions of tokens are considered, it
is unclear how scalable is curation and whether we will run out of unique
high-quality data soon. At variance with previous beliefs, we show that
properly filtered and deduplicated web data alone can lead to powerful models;
even significantly outperforming models from the state-of-the-art trained on
The Pile. Despite extensive filtering, the high-quality data we extract from
the web is still plentiful, and we are able to obtain five trillion tokens from
CommonCrawl. We publicly release an extract of 600 billion tokens from our
RefinedWeb dataset, and 1.3/7.5B parameters language models trained on it.
Filtered and deduplicated web data alone can produce powerful large language models that outperform those trained on The Pile.