FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data
  Processing to Every Language
Pre-training state-of-the-art large language models (LLMs) requires vast
amounts of clean and diverse text data. While the open development of large
high-quality English pre-training datasets has seen substantial recent
progress, training performant multilingual LLMs remains a challenge, in large
part due to the inherent difficulty of tailoring filtering and deduplication
pipelines to a large number of languages. In this work, we introduce a new
pre-training dataset curation pipeline based on FineWeb that can be
automatically adapted to support any language. We extensively ablate our
pipeline design choices on a set of nine diverse languages, guided by a set of
meaningful and informative evaluation tasks that were chosen through a novel
selection process based on measurable criteria. Ultimately, we show that our
pipeline can be used to create non-English corpora that produce more performant
models than prior datasets. We additionally introduce a straightforward and
principled approach to rebalance datasets that takes into consideration both
duplication count and quality, providing an additional performance uplift.
Finally, we scale our pipeline to over 1000 languages using almost 100 Common
Crawl snapshots to produce FineWeb2, a new 20 terabyte (5 billion document)
multilingual dataset which we release along with our pipeline, training, and
evaluation codebases.
685de28771131fa43be08a6c
Guilherme Penedo
admin_assigned
2025-06-27T09:07:07.724Z
62596f9e1c0a084224b93e00
https://cdn-avatars.huggingface.co/v1/production/uploads/62596f9e1c0a084224b93e00/X2aLkJ0ofhkXwAg7lXvxD.jpeg
Guilherme Penedo
user
guipenedo
685de28771131fa43be08a6d
Hynek Kydlíček
admin_assigned
2025-06-27T09:13:44.375Z
626ede24d2fa9e7d598c8709
https://cdn-avatars.huggingface.co/v1/production/uploads/626ede24d2fa9e7d598c8709/JKS8-Y2Jw87EgNQZBRswq.jpeg
Hynek Kydlicek
user
hynky
685de28771131fa43be08a6e
Vinko Sabolčec
admin_assigned
2025-06-27T09:13:50.541Z
65ba0b22eb364bc793936fc3
/avatars/1ca6827f5800620085746966b5f4c91f.svg
Vinko Sabolcec
user
vsabolcec
685de28771131fa43be08a6f
Bettina Messmer
claimed_verified
2025-07-13T07:29:57.538Z
65524059f71a0f5860d0aeaa
/avatars/c4297407e40dbe6088c2ae9d311f3f30.svg
Messmer
user
NXz64Fdf8Y
685de28771131fa43be08a70
Negar Foroutan
admin_assigned
2025-06-27T09:15:48.467Z
65b960eeeedfbaf37f8f6869
https://cdn-avatars.huggingface.co/v1/production/uploads/65b960eeeedfbaf37f8f6869/aVIaa_roB-j157KhhDaf2.jpeg
Negar Foroutan
user
negar-foroutan
685de28771131fa43be08a71
Amir Hossein Kargaran
admin_assigned
2025-06-27T09:15:55.282Z
61bf84c8ca59d6d196a1b4e8
https://cdn-avatars.huggingface.co/v1/production/uploads/61bf84c8ca59d6d196a1b4e8/L_NvUwlMYcye9X35z6f7e.jpeg
Amir Hossein Kargaran
user
kargaranamir
685de28771131fa43be08a72
Colin Raffel
admin_assigned
2025-06-27T09:16:04.207Z
6079c29765b9d0165cb18392
https://cdn-avatars.huggingface.co/v1/production/uploads/1618592397610-noauth.jpeg
Colin Raffel
user
craffel
685de28771131fa43be08a73
Martin Jaggi
admin_assigned
2025-06-27T09:16:10.905Z
654baf61d625e083383dfd00
/avatars/e933b2b9d7226e9bb72f77916c2be4f1.svg
Martin Jaggi
user
mjaggi
685de28771131fa43be08a74
Leandro Von Werra
admin_assigned
2025-06-27T09:16:18.216Z
5e48005437cb5b49818287a5
https://cdn-avatars.huggingface.co/v1/production/uploads/5e48005437cb5b49818287a5/4uCXGGui-9QifAT4qelxU.png
Leandro von Werra
user
lvwerra
685de28771131fa43be08a75
Thomas Wolf
admin_assigned
2025-06-27T09:16:30.072Z
5df7e9e5da6d0311fd3d53f9
https://cdn-avatars.huggingface.co/v1/production/uploads/1583857746553-5df7e9e5da6d0311fd3d53f9.jpeg
Thomas Wolf
user
thomwolf
FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data
  Processing to Every Language
Pre-training state-of-the-art large language models (LLMs) requires vast
amounts of clean and diverse text data. While the open development of large
high-quality English pre-training datasets has seen substantial recent
progress, training performant multilingual LLMs remains a challenge, in large
part due to the inherent difficulty of tailoring filtering and deduplication
pipelines to a large number of languages. In this work, we introduce a new
pre-training dataset curation pipeline based on FineWeb that can be
automatically adapted to support any language. We extensively ablate our
pipeline design choices on a set of nine diverse languages, guided by a set of
meaningful and informative evaluation tasks that were chosen through a novel
selection process based on measurable criteria. Ultimately, we show that our
pipeline can be used to create non-English corpora that produce more performant
models than prior datasets. We additionally introduce a straightforward and
principled approach to rebalance datasets that takes into consideration both
duplication count and quality, providing an additional performance uplift.
Finally, we scale our pipeline to over 1000 languages using almost 100 Common
Crawl snapshots to produce FineWeb2, a new 20 terabyte (5 billion document)
multilingual dataset which we release along with our pipeline, training, and
evaluation codebases.
A new pre-training dataset curation pipeline based on FineWeb supports multilingual LLMs by automatically adapting to any language, improving model performance and balancing dataset quality.
large language models pre-training datasets FineWeb multilingual LLMs filtering deduplication pipeline design evaluation tasks corpus creation dataset rebalancing FineWeb2 Common Crawl multilingual dataset