Craw4LLM: Efficient Web Crawling for LLM Pretraining
Web crawl is a main source of large language models' (LLMs) pretraining data,
but the majority of crawled web pages are discarded in pretraining due to low
data quality. This paper presents Crawl4LLM, an efficient web crawling method
that explores the web graph based on the preference of LLM pretraining.
Specifically, it leverages the influence of a webpage in LLM pretraining as the
priority score of the web crawler's scheduler, replacing the standard graph
connectivity based priority. Our experiments on a web graph containing 900
million webpages from a commercial search engine's index demonstrate the
efficiency of Crawl4LLM in obtaining high-quality pretraining data. With just
21% URLs crawled, LLMs pretrained on Crawl4LLM data reach the same downstream
performances of previous crawls, significantly reducing the crawling waste and
alleviating the burdens on websites. Our code is publicly available at
https://github.com/cxcscmu/Crawl4LLM.
67b6a7e83ef3656c48f149b9
Shi Yu
claimed_verified
2025-02-20T09:35:47.487Z
6135eeeb5bc6ecdf86b60f0d
/avatars/43cedcf20ab6b0801a662787400e1384.svg
Shi Yu
user
yushi
67b6a7e83ef3656c48f149ba
Zhiyuan Liu
admin_assigned
2025-02-20T16:09:12.683Z
6310a3cd531cc21f9e06de6a
https://cdn-avatars.huggingface.co/v1/production/uploads/6310a3cd531cc21f9e06de6a/aTGMx3O41lUARK9s3dAik.jpeg
Zhiyuan Liu
user
acharkq
67b6a7e83ef3656c48f149bb
Chenyan Xiong
admin_assigned
2025-02-20T16:09:19.014Z
617ae49bc0ff6006217aa22e
https://cdn-avatars.huggingface.co/v1/production/uploads/1635543252231-617ae49bc0ff6006217aa22e.jpeg
Chenyan Xiong
user
xiongchenyan
Craw4LLM: Efficient Web Crawling for LLM Pretraining
Web crawl is a main source of large language models' (LLMs) pretraining data,
but the majority of crawled web pages are discarded in pretraining due to low
data quality. This paper presents Crawl4LLM, an efficient web crawling method
that explores the web graph based on the preference of LLM pretraining.
Specifically, it leverages the influence of a webpage in LLM pretraining as the
priority score of the web crawler's scheduler, replacing the standard graph
connectivity based priority. Our experiments on a web graph containing 900
million webpages from a commercial search engine's index demonstrate the
efficiency of Crawl4LLM in obtaining high-quality pretraining data. With just
21% URLs crawled, LLMs pretrained on Crawl4LLM data reach the same downstream
performances of previous crawls, significantly reducing the crawling waste and
alleviating the burdens on websites. Our code is publicly available at
https://github.com/cxcscmu/Crawl4LLM.
Crawl4LLM is an efficient web crawling method that prioritizes webpages based on their influence in LLM pretraining, significantly reducing the amount of low-quality data and improving downstream performance.
LLMs large language models web crawling web graph LLM pretraining priority score web crawler's scheduler graph connectivity crawling waste downstream performance