title: Craw4LLM: Efficient Web Crawling for LLM Pretraining
summary: Web crawl is a main source of large language models' (LLMs) pretraining data,
  but the majority of crawled web pages are discarded in pretraining due to low
  data quality. This paper presents Crawl4LLM, an efficient web crawling method
  that explores the web graph based on the preference of LLM pretraining.
  Specifically, it leverages the influence of a webpage in LLM pretraining as the
  priority score of the web crawler's scheduler, replacing the standard graph
  connectivity based priority. Our experiments on a web graph containing 900
  million webpages from a commercial search engine's index demonstrate the
  efficiency of Crawl4LLM in obtaining high-quality pretraining data. With just
  21% URLs crawled, LLMs pretrained on Crawl4LLM data reach the same downstream
  performances of previous crawls, significantly reducing the crawling waste and
  alleviating the burdens on websites. Our code is publicly available at
  https://github.com/cxcscmu/Crawl4LLM.
paper_authors._id: 67b6a7e83ef3656c48f149b9
paper_authors.name: Shi Yu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-20T09:35:47.487Z
paper_authors._id: 6135eeeb5bc6ecdf86b60f0d
paper_authors.avatarUrl: /avatars/43cedcf20ab6b0801a662787400e1384.svg
paper_authors.fullname: Shi Yu
paper_authors.type: user
paper_authors.user: yushi
paper_authors._id: 67b6a7e83ef3656c48f149ba
paper_authors.name: Zhiyuan Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-20T16:09:12.683Z
paper_authors._id: 6310a3cd531cc21f9e06de6a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6310a3cd531cc21f9e06de6a/aTGMx3O41lUARK9s3dAik.jpeg
paper_authors.fullname: Zhiyuan Liu
paper_authors.type: user
paper_authors.user: acharkq
paper_authors._id: 67b6a7e83ef3656c48f149bb
paper_authors.name: Chenyan Xiong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-20T16:09:19.014Z
paper_authors._id: 617ae49bc0ff6006217aa22e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1635543252231-617ae49bc0ff6006217aa22e.jpeg
paper_authors.fullname: Chenyan Xiong
paper_authors.type: user
paper_authors.user: xiongchenyan
paper_title: Craw4LLM: Efficient Web Crawling for LLM Pretraining
paper_summary: Web crawl is a main source of large language models' (LLMs) pretraining data,
  but the majority of crawled web pages are discarded in pretraining due to low
  data quality. This paper presents Crawl4LLM, an efficient web crawling method
  that explores the web graph based on the preference of LLM pretraining.
  Specifically, it leverages the influence of a webpage in LLM pretraining as the
  priority score of the web crawler's scheduler, replacing the standard graph
  connectivity based priority. Our experiments on a web graph containing 900
  million webpages from a commercial search engine's index demonstrate the
  efficiency of Crawl4LLM in obtaining high-quality pretraining data. With just
  21% URLs crawled, LLMs pretrained on Crawl4LLM data reach the same downstream
  performances of previous crawls, significantly reducing the crawling waste and
  alleviating the burdens on websites. Our code is publicly available at
  https://github.com/cxcscmu/Crawl4LLM.
paper_ai_summary: Crawl4LLM is an efficient web crawling method that prioritizes webpages based on their influence in LLM pretraining, significantly reducing the amount of low-quality data and improving downstream performance.
paper_ai_keywords: LLMs large language models web crawling web graph LLM pretraining priority score web crawler's scheduler graph connectivity crawling waste downstream performance