title: Programming Every Example: Lifting Pre-training Data Quality like
    Experts at Scale
summary: Large language model pre-training has traditionally relied on human experts
  to craft heuristics for improving the corpora quality, resulting in numerous
  rules developed to date. However, these rules lack the flexibility to address
  the unique characteristics of individual example effectively. Meanwhile,
  applying tailored rules to every example is impractical for human experts. In
  this paper, we demonstrate that even small language models, with as few as 0.3B
  parameters, can exhibit substantial data refining capabilities comparable to
  those of human experts. We introduce Programming Every Example (ProX), a novel
  framework that treats data refinement as a programming task, enabling models to
  refine corpora by generating and executing fine-grained operations, such as
  string normalization, for each individual example at scale. Experimental
  results show that models pre-trained on ProX-curated data outperform either
  original data or data filtered by other selection methods by more than 2%
  across various downstream benchmarks. Its effectiveness spans various model
  sizes and pre-training corpora, including C4, RedPajama-V2, and FineWeb.
  Furthermore, ProX exhibits significant potential in domain-specific continual
  pre-training: without domain specific design, models trained on OpenWebMath
  refined by ProX outperform human-crafted rule-based methods, improving average
  accuracy by 7.6% over Mistral-7B, with 14.6% for Llama-2-7B and 20.3% for
  CodeLlama-7B, all within 10B tokens to be comparable to models like Llemma-7B
  trained on 200B tokens. Further analysis highlights that ProX significantly
  saves training FLOPs, offering a promising path for efficient LLM
  pre-training.We are open-sourcing ProX with >100B corpus, models, and sharing
  all training and implementation details for reproducible research and future
  innovation. Code: https://github.com/GAIR-NLP/ProX
paper_authors._id: 66f4c2f7529e23b71ff72e1e
paper_authors.name: Fan Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-26T08:23:08.164Z
paper_authors._id: 628f6e5ab90dde28ef57d293
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/628f6e5ab90dde28ef57d293/AxNzR2nvrND6Rf3RPkYMk.jpeg
paper_authors.fullname: Fan Zhou
paper_authors.type: user
paper_authors.user: koalazf99
paper_authors._id: 66f4c2f7529e23b71ff72e1f
paper_authors.name: Zengzhi Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-26T08:41:23.506Z
paper_authors._id: 62cbeb2d72dfd24b86bdf977
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62cbeb2d72dfd24b86bdf977/UcGYYSBNrCvPM5K9v-sro.png
paper_authors.fullname: Zengzhi Wang
paper_authors.type: user
paper_authors.user: SinclairWang
paper_authors._id: 66f4c2f7529e23b71ff72e20
paper_authors.name: Qian Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-30T08:07:20.519Z
paper_authors._id: 612ee6a7b960e78c6d2319d4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/612ee6a7b960e78c6d2319d4/2Hu9BaAyXbyh1vt0v1Qui.jpeg
paper_authors.fullname: Qian Liu
paper_authors.type: user
paper_authors.user: SivilTaram
paper_authors._id: 66f4c2f7529e23b71ff72e21
paper_authors.name: Junlong Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-26T08:42:06.216Z
paper_authors._id: 621e40ac944c7e36aaec2369
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/621e40ac944c7e36aaec2369/Yj-FJRWps3rvsS_B2bnKo.jpeg
paper_authors.fullname: Junlong Li
paper_authors.type: user
paper_authors.user: lockon
paper_authors._id: 66f4c2f7529e23b71ff72e22
paper_authors.name: Pengfei Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-26T08:42:23.621Z
paper_authors._id: 6144a0c4ff1146bbd84d9865
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1661715958139-6144a0c4ff1146bbd84d9865.png
paper_authors.fullname: Pengfei Liu
paper_authors.type: user
paper_authors.user: Pengfei
paper_title: Programming Every Example: Lifting Pre-training Data Quality like
    Experts at Scale
paper_summary: Large language model pre-training has traditionally relied on human experts
  to craft heuristics for improving the corpora quality, resulting in numerous
  rules developed to date. However, these rules lack the flexibility to address
  the unique characteristics of individual example effectively. Meanwhile,
  applying tailored rules to every example is impractical for human experts. In
  this paper, we demonstrate that even small language models, with as few as 0.3B
  parameters, can exhibit substantial data refining capabilities comparable to
  those of human experts. We introduce Programming Every Example (ProX), a novel
  framework that treats data refinement as a programming task, enabling models to
  refine corpora by generating and executing fine-grained operations, such as
  string normalization, for each individual example at scale. Experimental
  results show that models pre-trained on ProX-curated data outperform either
  original data or data filtered by other selection methods by more than 2%
  across various downstream benchmarks. Its effectiveness spans various model
  sizes and pre-training corpora, including C4, RedPajama-V2, and FineWeb.
  Furthermore, ProX exhibits significant potential in domain-specific continual
  pre-training: without domain specific design, models trained on OpenWebMath
  refined by ProX outperform human-crafted rule-based methods, improving average
  accuracy by 7.6% over Mistral-7B, with 14.6% for Llama-2-7B and 20.3% for
  CodeLlama-7B, all within 10B tokens to be comparable to models like Llemma-7B
  trained on 200B tokens. Further analysis highlights that ProX significantly
  saves training FLOPs, offering a promising path for efficient LLM
  pre-training.We are open-sourcing ProX with >100B corpus, models, and sharing
  all training and implementation details for reproducible research and future
  innovation. Code: https://github.com/GAIR-NLP/ProX
paper_ai_summary: ProX, a novel framework treating data refinement as a programming task, enhances large language model pre-training by generating fine-grained operations for each example, outperforming traditional human-crafted rule-based methods significantly across various benchmarks and domains.
paper_ai_keywords: Programming Every Example (ProX) data refinement string normalization LLLM pre-training C4 RedPajama-V2 FineWeb domain-specific continual pre-training OpenWebMath Mistral-7B Llama-2-7B CodeLlama-7B training FLOPs