title: The FineWeb Datasets: Decanting the Web for the Finest Text Data at
    Scale
summary: The performance of a large language model (LLM) depends heavily on the
  quality and size of its pretraining dataset. However, the pretraining datasets
  for state-of-the-art open LLMs like Llama 3 and Mixtral are not publicly
  available and very little is known about how they were created. In this work,
  we introduce FineWeb, a 15-trillion token dataset derived from 96 Common Crawl
  snapshots that produces better-performing LLMs than other open pretraining
  datasets. To advance the understanding of how best to curate high-quality
  pretraining datasets, we carefully document and ablate all of the design
  choices used in FineWeb, including in-depth investigations of deduplication and
  filtering strategies. In addition, we introduce FineWeb-Edu, a 1.3-trillion
  token collection of educational text filtered from FineWeb. LLMs pretrained on
  FineWeb-Edu exhibit dramatically better performance on knowledge- and
  reasoning-intensive benchmarks like MMLU and ARC. Along with our datasets, we
  publicly release our data curation codebase and all of the models trained
  during our ablation experiments.
paper_authors._id: 667baf1709faa9d48c05b503
paper_authors.name: Guilherme Penedo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-26T08:03:02.663Z
paper_authors._id: 62596f9e1c0a084224b93e00
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62596f9e1c0a084224b93e00/X2aLkJ0ofhkXwAg7lXvxD.jpeg
paper_authors.fullname: Guilherme Penedo
paper_authors.type: user
paper_authors.user: guipenedo
paper_authors._id: 667baf1709faa9d48c05b504
paper_authors.name: Hynek Kydlíček
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-26T09:53:11.720Z
paper_authors._id: 626ede24d2fa9e7d598c8709
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/626ede24d2fa9e7d598c8709/JKS8-Y2Jw87EgNQZBRswq.jpeg
paper_authors.fullname: Hynek Kydlicek
paper_authors.type: user
paper_authors.user: hynky
paper_authors._id: 667baf1709faa9d48c05b505
paper_authors.name: Loubna Ben allal
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-26T08:36:05.109Z
paper_authors._id: 61c141342aac764ce1654e43
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61c141342aac764ce1654e43/81AwoT5IQ_Xdw0OVw7TKu.jpeg
paper_authors.fullname: Loubna Ben Allal
paper_authors.type: user
paper_authors.user: loubnabnl
paper_authors._id: 667baf1709faa9d48c05b506
paper_authors.name: Anton Lozhkov
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-26T09:53:20.691Z
paper_authors._id: 602e6dee60e3dd96631c906e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1613655355830-noauth.png
paper_authors.fullname: Anton Lozhkov
paper_authors.type: user
paper_authors.user: anton-l
paper_authors._id: 667baf1709faa9d48c05b507
paper_authors.name: Margaret Mitchell
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-26T09:53:30.450Z
paper_authors._id: 60c757ea5f9a76ab3f844f12
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1626214544196-60c757ea5f9a76ab3f844f12.png
paper_authors.fullname: Margaret Mitchell
paper_authors.type: user
paper_authors.user: meg
paper_authors._id: 667baf1709faa9d48c05b508
paper_authors.name: Colin Raffel
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-26T09:53:36.838Z
paper_authors._id: 6079c29765b9d0165cb18392
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1618592397610-noauth.jpeg
paper_authors.fullname: Colin Raffel
paper_authors.type: user
paper_authors.user: craffel
paper_authors._id: 667baf1709faa9d48c05b509
paper_authors.name: Leandro Von Werra
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-26T09:53:42.769Z
paper_authors._id: 5e48005437cb5b49818287a5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5e48005437cb5b49818287a5/4uCXGGui-9QifAT4qelxU.png
paper_authors.fullname: Leandro von Werra
paper_authors.type: user
paper_authors.user: lvwerra
paper_authors._id: 667baf1709faa9d48c05b50a
paper_authors.name: Thomas Wolf
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-26T09:53:49.151Z
paper_authors._id: 5df7e9e5da6d0311fd3d53f9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1583857746553-5df7e9e5da6d0311fd3d53f9.jpeg
paper_authors.fullname: Thomas Wolf
paper_authors.type: user
paper_authors.user: thomwolf
paper_title: The FineWeb Datasets: Decanting the Web for the Finest Text Data at
    Scale
paper_summary: The performance of a large language model (LLM) depends heavily on the
  quality and size of its pretraining dataset. However, the pretraining datasets
  for state-of-the-art open LLMs like Llama 3 and Mixtral are not publicly
  available and very little is known about how they were created. In this work,
  we introduce FineWeb, a 15-trillion token dataset derived from 96 Common Crawl
  snapshots that produces better-performing LLMs than other open pretraining
  datasets. To advance the understanding of how best to curate high-quality
  pretraining datasets, we carefully document and ablate all of the design
  choices used in FineWeb, including in-depth investigations of deduplication and
  filtering strategies. In addition, we introduce FineWeb-Edu, a 1.3-trillion
  token collection of educational text filtered from FineWeb. LLMs pretrained on
  FineWeb-Edu exhibit dramatically better performance on knowledge- and
  reasoning-intensive benchmarks like MMLU and ARC. Along with our datasets, we
  publicly release our data curation codebase and all of the models trained
  during our ablation experiments.
paper_ai_summary: FineWeb, a 15-trillion token dataset from Common Crawl snapshots, outperforms other open datasets in LLM training, and its educational subset, FineWeb-Edu, significantly improves performance on knowledge and reasoning benchmarks.
paper_ai_keywords: large language model pretraining dataset FineWeb Common Crawl deduplication filtering strategies FineWeb-Edu MMLU ARC