The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
<|reference_start|>The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale: The performance of a large language model (LLM) depends heavily on the quality and size of its pretraining dataset. However, the pretraining datasets for state-of-the-art open LLMs like Llama 3 and Mixtral are not publicly available and very little is known about how they were created. In this work, we introduce FineWeb, a 15-trillion token dataset derived from 96 Common Crawl snapshots that produces better-performing LLMs than other open pretraining datasets. To advance the understanding of how best to curate high-quality pretraining datasets, we carefully document and ablate all of the design choices used in FineWeb, including in-depth investigations of deduplication and filtering strategies. In addition, we introduce FineWeb-Edu, a 1.3-trillion token collection of educational text filtered from FineWeb. LLMs pretrained on FineWeb-Edu exhibit dramatically better performance on knowledge- and reasoning-intensive benchmarks like MMLU and ARC. Along with our datasets, we publicly release our data curation codebase and all of the models trained during our ablation experiments.<|reference_end|>
@article{penedo2024the,
    title={The FineWeb Datasets: Decanting the Web for the Finest Text Data at
  Scale},
    author={Guilherme Penedo, Hynek Kydl'iv{c}ek, Loubna Ben allal, Anton
  Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, Thomas Wolf},
    journal={arXiv preprint arXiv:2406.17557},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.17557},
    primaryClass={cs.CL}
}