title: Data Selection via Optimal Control for Language Models
summary: This work investigates the selection of high-quality pre-training data from
  massive corpora to enhance LMs' capabilities for downstream usage. We formulate
  data selection as a generalized Optimal Control problem, which can be solved
  theoretically by Pontryagin's Maximum Principle (PMP), yielding a set of
  necessary conditions that characterize the relationship between optimal data
  selection and LM training dynamics. Based on these theoretical results, we
  introduce PMP-based Data Selection (PDS), a framework that approximates optimal
  data selection by solving the PMP conditions. In our experiments, we adopt PDS
  to select data from CommmonCrawl and show that the PDS-selected corpus
  accelerates the learning of LMs and constantly boosts their performance on a
  wide range of downstream tasks across various model sizes. Moreover, the
  benefits of PDS extend to ~400B models trained on ~10T tokens, as evidenced by
  the extrapolation of the test loss curves according to the Scaling Laws. PDS
  also improves data utilization when the pre-training data is limited, by
  reducing the data demand by 1.8 times, which mitigates the quick exhaustion of
  available web-crawled corpora. Our code, data, and model checkpoints can be
  found in https://github.com/microsoft/LMOps/tree/main/data_selection.
paper_authors._id: 670739dfd4311c2dd68f7fa5
paper_authors.name: Yuxian Gu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-10T08:01:31.000Z
paper_authors._id: 624ac662102fcdff87be51b9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/624ac662102fcdff87be51b9/rzNahZFFkp194170tactJ.jpeg
paper_authors.fullname: Yuxian Gu
paper_authors.type: user
paper_authors.user: t1101675
paper_authors._id: 670739dfd4311c2dd68f7fa6
paper_authors.name: Li Dong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-11T07:44:21.119Z
paper_authors._id: 5df85abada6d0311fd3d5408
paper_authors.avatarUrl: /avatars/2331cf703c1b5d3a62e2050b1a6eb108.svg
paper_authors.fullname: Li Dong
paper_authors.type: user
paper_authors.user: unilm
paper_authors._id: 670739dfd4311c2dd68f7fa7
paper_authors.name: Hongning Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T09:25:41.531Z
paper_authors._id: 62ec23fcbd19e355478fc584
paper_authors.avatarUrl: /avatars/d7c567ef5f20bb3b9905cb5015d11e12.svg
paper_authors.fullname: Hongning Wang
paper_authors.type: user
paper_authors.user: howang
paper_authors._id: 670739dfd4311c2dd68f7fa8
paper_authors.name: Yaru Hao
paper_authors._id: 670739dfd4311c2dd68f7fa9
paper_authors.name: Qingxiu Dong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-10T08:01:32.592Z
paper_authors._id: 670740744341dcee459fb990
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/66UkZvrAk7fQr5YCylEFk.png
paper_authors.fullname: Qingxiu Dong
paper_authors.type: user
paper_authors.user: Rsy24
paper_authors._id: 670739dfd4311c2dd68f7faa
paper_authors.name: Furu Wei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T09:25:28.584Z
paper_authors._id: 6368c512fbfe97c16a40baba
paper_authors.avatarUrl: /avatars/1c23bc7c0b6d9225699ce27647623d7a.svg
paper_authors.fullname: Furu Wei
paper_authors.type: user
paper_authors.user: thegenerality
paper_authors._id: 670739dfd4311c2dd68f7fab
paper_authors.name: Minlie Huang
paper_title: Data Selection via Optimal Control for Language Models
paper_summary: This work investigates the selection of high-quality pre-training data from
  massive corpora to enhance LMs' capabilities for downstream usage. We formulate
  data selection as a generalized Optimal Control problem, which can be solved
  theoretically by Pontryagin's Maximum Principle (PMP), yielding a set of
  necessary conditions that characterize the relationship between optimal data
  selection and LM training dynamics. Based on these theoretical results, we
  introduce PMP-based Data Selection (PDS), a framework that approximates optimal
  data selection by solving the PMP conditions. In our experiments, we adopt PDS
  to select data from CommmonCrawl and show that the PDS-selected corpus
  accelerates the learning of LMs and constantly boosts their performance on a
  wide range of downstream tasks across various model sizes. Moreover, the
  benefits of PDS extend to ~400B models trained on ~10T tokens, as evidenced by
  the extrapolation of the test loss curves according to the Scaling Laws. PDS
  also improves data utilization when the pre-training data is limited, by
  reducing the data demand by 1.8 times, which mitigates the quick exhaustion of
  available web-crawled corpora. Our code, data, and model checkpoints can be
  found in https://github.com/microsoft/LMOps/tree/main/data_selection.
paper_ai_keywords: Optimal Control problem Pontryagin's Maximum Principle (PMP) PMP-based Data Selection (PDS) CommmonCrawl Scaling Laws test loss curves data utilization data demand