The FineWeb Datasets: Decanting the Web for the Finest Text Data at
  Scale
The performance of a large language model (LLM) depends heavily on the
quality and size of its pretraining dataset. However, the pretraining datasets
for state-of-the-art open LLMs like Llama 3 and Mixtral are not publicly
available and very little is known about how they were created. In this work,
we introduce FineWeb, a 15-trillion token dataset derived from 96 Common Crawl
snapshots that produces better-performing LLMs than other open pretraining
datasets. To advance the understanding of how best to curate high-quality
pretraining datasets, we carefully document and ablate all of the design
choices used in FineWeb, including in-depth investigations of deduplication and
filtering strategies. In addition, we introduce FineWeb-Edu, a 1.3-trillion
token collection of educational text filtered from FineWeb. LLMs pretrained on
FineWeb-Edu exhibit dramatically better performance on knowledge- and
reasoning-intensive benchmarks like MMLU and ARC. Along with our datasets, we
publicly release our data curation codebase and all of the models trained
during our ablation experiments.
667baf1709faa9d48c05b503
Guilherme Penedo
claimed_verified
2024-06-26T08:03:02.663Z
62596f9e1c0a084224b93e00
https://cdn-avatars.huggingface.co/v1/production/uploads/62596f9e1c0a084224b93e00/X2aLkJ0ofhkXwAg7lXvxD.jpeg
Guilherme Penedo
user
guipenedo
667baf1709faa9d48c05b504
Hynek Kydlíček
admin_assigned
2024-06-26T09:53:11.720Z
626ede24d2fa9e7d598c8709
https://cdn-avatars.huggingface.co/v1/production/uploads/626ede24d2fa9e7d598c8709/JKS8-Y2Jw87EgNQZBRswq.jpeg
Hynek Kydlicek
user
hynky
667baf1709faa9d48c05b505
Loubna Ben allal
claimed_verified
2024-06-26T08:36:05.109Z
61c141342aac764ce1654e43
https://cdn-avatars.huggingface.co/v1/production/uploads/61c141342aac764ce1654e43/81AwoT5IQ_Xdw0OVw7TKu.jpeg
Loubna Ben Allal
user
loubnabnl
667baf1709faa9d48c05b506
Anton Lozhkov
admin_assigned
2024-06-26T09:53:20.691Z
602e6dee60e3dd96631c906e
https://cdn-avatars.huggingface.co/v1/production/uploads/1613655355830-noauth.png
Anton Lozhkov
user
anton-l
667baf1709faa9d48c05b507
Margaret Mitchell
admin_assigned
2024-06-26T09:53:30.450Z
60c757ea5f9a76ab3f844f12
https://cdn-avatars.huggingface.co/v1/production/uploads/1626214544196-60c757ea5f9a76ab3f844f12.png
Margaret Mitchell
user
meg
667baf1709faa9d48c05b508
Colin Raffel
admin_assigned
2024-06-26T09:53:36.838Z
6079c29765b9d0165cb18392
https://cdn-avatars.huggingface.co/v1/production/uploads/1618592397610-noauth.jpeg
Colin Raffel
user
craffel
667baf1709faa9d48c05b509
Leandro Von Werra
admin_assigned
2024-06-26T09:53:42.769Z
5e48005437cb5b49818287a5
https://cdn-avatars.huggingface.co/v1/production/uploads/5e48005437cb5b49818287a5/4uCXGGui-9QifAT4qelxU.png
Leandro von Werra
user
lvwerra
667baf1709faa9d48c05b50a
Thomas Wolf
admin_assigned
2024-06-26T09:53:49.151Z
5df7e9e5da6d0311fd3d53f9
https://cdn-avatars.huggingface.co/v1/production/uploads/1583857746553-5df7e9e5da6d0311fd3d53f9.jpeg
Thomas Wolf
user
thomwolf
The FineWeb Datasets: Decanting the Web for the Finest Text Data at
  Scale
The performance of a large language model (LLM) depends heavily on the
quality and size of its pretraining dataset. However, the pretraining datasets
for state-of-the-art open LLMs like Llama 3 and Mixtral are not publicly
available and very little is known about how they were created. In this work,
we introduce FineWeb, a 15-trillion token dataset derived from 96 Common Crawl
snapshots that produces better-performing LLMs than other open pretraining
datasets. To advance the understanding of how best to curate high-quality
pretraining datasets, we carefully document and ablate all of the design
choices used in FineWeb, including in-depth investigations of deduplication and
filtering strategies. In addition, we introduce FineWeb-Edu, a 1.3-trillion
token collection of educational text filtered from FineWeb. LLMs pretrained on
FineWeb-Edu exhibit dramatically better performance on knowledge- and
reasoning-intensive benchmarks like MMLU and ARC. Along with our datasets, we
publicly release our data curation codebase and all of the models trained
during our ablation experiments.
FineWeb, a 15-trillion token dataset from Common Crawl snapshots, outperforms other open datasets in LLM training, and its educational subset, FineWeb-Edu, significantly improves performance on knowledge and reasoning benchmarks.
large language model pretraining dataset FineWeb Common Crawl deduplication filtering strategies FineWeb-Edu MMLU ARC