title: The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora
    with Web Data, and Web Data Only
summary: Large language models are commonly trained on a mixture of filtered web data
  and curated high-quality corpora, such as social media conversations, books, or
  technical papers. This curation process is believed to be necessary to produce
  performant models with broad zero-shot generalization abilities. However, as
  larger models requiring pretraining on trillions of tokens are considered, it
  is unclear how scalable is curation and whether we will run out of unique
  high-quality data soon. At variance with previous beliefs, we show that
  properly filtered and deduplicated web data alone can lead to powerful models;
  even significantly outperforming models from the state-of-the-art trained on
  The Pile. Despite extensive filtering, the high-quality data we extract from
  the web is still plentiful, and we are able to obtain five trillion tokens from
  CommonCrawl. We publicly release an extract of 600 billion tokens from our
  RefinedWeb dataset, and 1.3/7.5B parameters language models trained on it.
paper_authors._id: 647d5098396de7684dec2d0b
paper_authors.name: Guilherme Penedo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-06T08:39:51.360Z
paper_authors._id: 62596f9e1c0a084224b93e00
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62596f9e1c0a084224b93e00/X2aLkJ0ofhkXwAg7lXvxD.jpeg
paper_authors.fullname: Guilherme Penedo
paper_authors.type: user
paper_authors.user: guipenedo
paper_authors._id: 647d5098396de7684dec2d0c
paper_authors.name: Quentin Malartic
paper_authors._id: 647d5098396de7684dec2d0d
paper_authors.name: Daniel Hesslow
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-05T16:38:54.213Z
paper_authors._id: 61a8d1aac664736898ffc84f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1654022901910-61a8d1aac664736898ffc84f.jpeg
paper_authors.fullname: Daniel Hesslow
paper_authors.type: user
paper_authors.user: DanielHesslow
paper_authors._id: 647d5098396de7684dec2d0e
paper_authors.name: Ruxandra Cojocaru
paper_authors._id: 647d5098396de7684dec2d0f
paper_authors.name: Alessandro Cappelli
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-07T10:09:45.941Z
paper_authors._id: 647edf57d26579210f58bcf3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/647edf57d26579210f58bcf3/jyZbIiYxcS6s_i9FbDxJv.jpeg
paper_authors.fullname: alessandro cappelli
paper_authors.type: user
paper_authors.user: alecap
paper_authors._id: 647d5098396de7684dec2d10
paper_authors.name: Hamza Alobeidli
paper_authors._id: 647d5098396de7684dec2d11
paper_authors.name: Baptiste Pannier
paper_authors._id: 647d5098396de7684dec2d12
paper_authors.name: Ebtesam Almazrouei
paper_authors._id: 647d5098396de7684dec2d13
paper_authors.name: Julien Launay
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-06T08:39:54.455Z
paper_authors._id: 60938bb4c65685640e32a7df
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1620282175694-noauth.png
paper_authors.fullname: Julien Launay
paper_authors.type: user
paper_authors.user: slippylolo
paper_title: The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora
    with Web Data, and Web Data Only
paper_summary: Large language models are commonly trained on a mixture of filtered web data
  and curated high-quality corpora, such as social media conversations, books, or
  technical papers. This curation process is believed to be necessary to produce
  performant models with broad zero-shot generalization abilities. However, as
  larger models requiring pretraining on trillions of tokens are considered, it
  is unclear how scalable is curation and whether we will run out of unique
  high-quality data soon. At variance with previous beliefs, we show that
  properly filtered and deduplicated web data alone can lead to powerful models;
  even significantly outperforming models from the state-of-the-art trained on
  The Pile. Despite extensive filtering, the high-quality data we extract from
  the web is still plentiful, and we are able to obtain five trillion tokens from
  CommonCrawl. We publicly release an extract of 600 billion tokens from our
  RefinedWeb dataset, and 1.3/7.5B parameters language models trained on it.
paper_ai_keywords: large language models pretraining filtered web data curated high-quality corpora social media conversations technical papers zero-shot generalization deduplicated web data CommonCrawl RefinedWeb dataset parameter-efficient fine-tuning