title: Predictive Data Selection: The Data That Predicts Is the Data That
    Teaches
summary: Language model pretraining involves training on extensive corpora, where data
  quality plays a pivotal role. In this work, we aim to directly estimate the
  contribution of data during pretraining and select pretraining data in an
  efficient manner. Specifically, we draw inspiration from recent findings
  showing that compression efficiency (i.e., the normalized loss) of diverse
  models on certain text correlates strongly with their downstream performance,
  when the text domain aligns with the downstream benchmark (Huang et al., 2024).
  Building on this observation, we hypothesize that data on which model losses
  are predictive of downstream abilities also contribute effectively to learning.
  To leverage this insight, we introduce data selection based on data's
  Predictive strength (Preselect), a lightweight and efficient data selection
  method that requires training and deploying only a fastText-based scorer.
  Through comprehensive experiments with 1B and 3B parameter models, we
  demonstrate that models trained on 30B tokens selected with PreSelect surpasses
  the performance of a vanilla baseline trained on 300B tokens, achieving a 10x
  reduction in compute requirements. Furthermore, PreSelect significantly
  outperforms other competitive data selection baselines, such as DCLM and
  FineWeb-Edu on a scale of 3B models trained on 100B tokens. We open-source our
  trained data selection scorer along with the curated datasets at
  https://github.com/hkust-nlp/PreSelect.
paper_authors._id: 67c66382e5394bda7cbd03f9
paper_authors.name: Kashun Shum
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-04T08:51:25.484Z
paper_authors._id: 641c9662043963b1c0a1df52
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/641c9662043963b1c0a1df52/L1o85EHztv_xP9r6ppljf.jpeg
paper_authors.fullname: KaShun SHUM
paper_authors.type: user
paper_authors.user: ksshumab
paper_authors._id: 67c66382e5394bda7cbd03fa
paper_authors.name: Yuzhen Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-04T08:51:23.329Z
paper_authors._id: 6462def82a83863b97c0611e
paper_authors.avatarUrl: /avatars/c03e9cc7d75b0266fcc56ecb6ee62148.svg
paper_authors.fullname: Yuzhen Huang
paper_authors.type: user
paper_authors.user: yuzhen17
paper_authors._id: 67c66382e5394bda7cbd03fb
paper_authors.name: Hongjian Zou
paper_authors._id: 67c66382e5394bda7cbd03fc
paper_authors.name: Ding Qi
paper_authors._id: 67c66382e5394bda7cbd03fd
paper_authors.name: Yixuan Liao
paper_authors._id: 67c66382e5394bda7cbd03fe
paper_authors.name: Xiaoxin Chen
paper_authors._id: 67c66382e5394bda7cbd03ff
paper_authors.name: Qian Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-27T09:05:18.358Z
paper_authors._id: 612ee6a7b960e78c6d2319d4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/612ee6a7b960e78c6d2319d4/2Hu9BaAyXbyh1vt0v1Qui.jpeg
paper_authors.fullname: Qian Liu
paper_authors.type: user
paper_authors.user: SivilTaram
paper_authors._id: 67c66382e5394bda7cbd0400
paper_authors.name: Junxian He
paper_title: Predictive Data Selection: The Data That Predicts Is the Data That
    Teaches
paper_summary: Language model pretraining involves training on extensive corpora, where data
  quality plays a pivotal role. In this work, we aim to directly estimate the
  contribution of data during pretraining and select pretraining data in an
  efficient manner. Specifically, we draw inspiration from recent findings
  showing that compression efficiency (i.e., the normalized loss) of diverse
  models on certain text correlates strongly with their downstream performance,
  when the text domain aligns with the downstream benchmark (Huang et al., 2024).
  Building on this observation, we hypothesize that data on which model losses
  are predictive of downstream abilities also contribute effectively to learning.
  To leverage this insight, we introduce data selection based on data's
  Predictive strength (Preselect), a lightweight and efficient data selection
  method that requires training and deploying only a fastText-based scorer.
  Through comprehensive experiments with 1B and 3B parameter models, we
  demonstrate that models trained on 30B tokens selected with PreSelect surpasses
  the performance of a vanilla baseline trained on 300B tokens, achieving a 10x
  reduction in compute requirements. Furthermore, PreSelect significantly
  outperforms other competitive data selection baselines, such as DCLM and
  FineWeb-Edu on a scale of 3B models trained on 100B tokens. We open-source our
  trained data selection scorer along with the curated datasets at
  https://github.com/hkust-nlp/PreSelect.
paper_ai_summary: A lightweight data selection method called PreSelect effectively reduces computational requirements while enhancing language model performance by focusing on data predictive of downstream abilities.
paper_ai_keywords: compression efficiency downstream performance data selection fastText predictive strength DCLM FineWeb-Edu