title: When Less is More: Investigating Data Pruning for Pretraining LLMs at
    Scale
summary: Large volumes of text data have contributed significantly to the development
  of large language models (LLMs) in recent years. This data is typically
  acquired by scraping the internet, leading to pretraining datasets comprised of
  noisy web text. To date, efforts to prune these datasets down to a higher
  quality subset have relied on hand-crafted heuristics encoded as rule-based
  filters. In this work, we take a wider view and explore scalable estimates of
  data quality that can be used to systematically measure the quality of
  pretraining data. We perform a rigorous comparison at scale of the simple data
  quality estimator of perplexity, as well as more sophisticated and
  computationally intensive estimates of the Error L2-Norm and memorization.
  These metrics are used to rank and prune pretraining corpora, and we
  subsequently compare LLMs trained on these pruned datasets. Surprisingly, we
  find that the simple technique of perplexity outperforms our more
  computationally expensive scoring methods. We improve over our no-pruning
  baseline while training on as little as 30% of the original training dataset.
  Our work sets the foundation for unexplored strategies in automatically
  curating high quality corpora and suggests the majority of pretraining data can
  be removed while retaining performance.
paper_authors._id: 64ffc6723587d3ebfd8041be
paper_authors.name: Max Marion
paper_authors.statusLastChangedAt: 2023-09-12T11:02:21.259Z
paper_authors._id: 64ffc6723587d3ebfd8041bf
paper_authors.name: Ahmet Üstün
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-12T11:02:36.732Z
paper_authors._id: 60d35d7ad7b174177faabd5b
paper_authors.avatarUrl: /avatars/4e5403b9d4a845a2d21e8217dc3c16d2.svg
paper_authors.fullname: Ahmet Üstün
paper_authors.type: user
paper_authors.user: ahmetu
paper_authors._id: 64ffc6723587d3ebfd8041c0
paper_authors.name: Luiza Pozzobon
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-12T11:02:49.169Z
paper_authors._id: 62740039e2be275e63b783e1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62740039e2be275e63b783e1/Ft9kE-JS4LPubVUiAgrv0.jpeg
paper_authors.fullname: Luiza Pozzobon
paper_authors.type: user
paper_authors.user: luizapzbn
paper_authors._id: 64ffc6723587d3ebfd8041c1
paper_authors.name: Alex Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-22T19:34:45.263Z
paper_authors._id: 618d96c9b8de35a67a79d288
paper_authors.avatarUrl: /avatars/3fbda3277052b5a2d34f84330c440e69.svg
paper_authors.fullname: Alex Wang
paper_authors.type: user
paper_authors.user: w4ngatang
paper_authors._id: 64ffc6723587d3ebfd8041c2
paper_authors.name: Marzieh Fadaee
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-21T12:06:26.345Z
paper_authors._id: 6441042d5d600fb0951a5f99
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6441042d5d600fb0951a5f99/4CbOaYcEz99BtVAQvnGTn.jpeg
paper_authors.fullname: Marzieh Fadaee
paper_authors.type: user
paper_authors.user: MarziehFadaee
paper_authors._id: 64ffc6723587d3ebfd8041c3
paper_authors.name: Sara Hooker
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-12T11:03:01.922Z
paper_authors._id: 63434eb76f59b79da07dbddf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63434eb76f59b79da07dbddf/BEwmVjqPNYlqmutXG0G6e.jpeg
paper_authors.fullname: Sara Hooker
paper_authors.type: user
paper_authors.user: sarahooker
paper_title: When Less is More: Investigating Data Pruning for Pretraining LLMs at
    Scale
paper_summary: Large volumes of text data have contributed significantly to the development
  of large language models (LLMs) in recent years. This data is typically
  acquired by scraping the internet, leading to pretraining datasets comprised of
  noisy web text. To date, efforts to prune these datasets down to a higher
  quality subset have relied on hand-crafted heuristics encoded as rule-based
  filters. In this work, we take a wider view and explore scalable estimates of
  data quality that can be used to systematically measure the quality of
  pretraining data. We perform a rigorous comparison at scale of the simple data
  quality estimator of perplexity, as well as more sophisticated and
  computationally intensive estimates of the Error L2-Norm and memorization.
  These metrics are used to rank and prune pretraining corpora, and we
  subsequently compare LLMs trained on these pruned datasets. Surprisingly, we
  find that the simple technique of perplexity outperforms our more
  computationally expensive scoring methods. We improve over our no-pruning
  baseline while training on as little as 30% of the original training dataset.
  Our work sets the foundation for unexplored strategies in automatically
  curating high quality corpora and suggests the majority of pretraining data can
  be removed while retaining performance.
paper_ai_keywords: large language models (LLMs) pretraining datasets noisy web text hand-crafted heuristics rule-based filters data quality estimator perplexity Error L2-Norm memorization pretraining corpora automatically curating high quality corpora