title: CoLoR-Filter: Conditional Loss Reduction Filtering for Targeted Language
    Model Pre-training
summary: Selecting high-quality data for pre-training is crucial in shaping the
  downstream task performance of language models. A major challenge lies in
  identifying this optimal subset, a problem generally considered intractable,
  thus necessitating scalable and effective heuristics. In this work, we propose
  a data selection method, CoLoR-Filter (Conditional Loss Reduction Filtering),
  which leverages an empirical Bayes-inspired approach to derive a simple and
  computationally efficient selection criterion based on the relative loss values
  of two auxiliary models.
    In addition to the modeling rationale, we evaluate CoLoR-Filter empirically
  on two language modeling tasks: (1) selecting data from C4 for domain
  adaptation to evaluation on Books and (2) selecting data from C4 for a suite of
  downstream multiple-choice question answering tasks. We demonstrate favorable
  scaling both as we subselect more aggressively and using small auxiliary models
  to select data for large target models. As one headline result, CoLoR-Filter
  data selected using a pair of 150m parameter auxiliary models can train a 1.2b
  parameter target model to match a 1.2b parameter model trained on 25b randomly
  selected tokens with 25x less data for Books and 11x less data for the
  downstream tasks.
    Code: https://github.com/davidbrandfonbrener/color-filter-olmo
    Filtered data:
  https://huggingface.co/datasets/davidbrandfonbrener/color-filtered-c4
paper_authors._id: 66716957a3f8ed647dd59dd1
paper_authors.name: David Brandfonbrener
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-18T16:29:03.692Z
paper_authors._id: 6314bdae4f283a6af50711da
paper_authors.avatarUrl: /avatars/ca3fcc8609b492124b4f97fab29a63f1.svg
paper_authors.fullname: David Brandfonbrener
paper_authors.type: user
paper_authors.user: davidbrandfonbrener
paper_authors._id: 66716957a3f8ed647dd59dd2
paper_authors.name: Hanlin Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-11T09:24:14.829Z
paper_authors._id: 624054bcc2c17da6a63eb539
paper_authors.avatarUrl: /avatars/bf52dc0683b4100733f8696a97696d0e.svg
paper_authors.fullname: hlzhang109
paper_authors.type: user
paper_authors.user: hlzhang109
paper_authors._id: 66716957a3f8ed647dd59dd3
paper_authors.name: Andreas Kirsch
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-18T16:29:24.042Z
paper_authors._id: 635831ee5ddda966186fe6a3
paper_authors.avatarUrl: /avatars/17aa4b17a4632af5409a31b23f1ecf34.svg
paper_authors.fullname: Andreas Kirsch
paper_authors.type: user
paper_authors.user: blackhc
paper_authors._id: 66716957a3f8ed647dd59dd4
paper_authors.name: Jonathan Richard Schwarz
paper_authors._id: 66716957a3f8ed647dd59dd5
paper_authors.name: Sham Kakade
paper_title: CoLoR-Filter: Conditional Loss Reduction Filtering for Targeted Language
    Model Pre-training
paper_summary: Selecting high-quality data for pre-training is crucial in shaping the
  downstream task performance of language models. A major challenge lies in
  identifying this optimal subset, a problem generally considered intractable,
  thus necessitating scalable and effective heuristics. In this work, we propose
  a data selection method, CoLoR-Filter (Conditional Loss Reduction Filtering),
  which leverages an empirical Bayes-inspired approach to derive a simple and
  computationally efficient selection criterion based on the relative loss values
  of two auxiliary models.
    In addition to the modeling rationale, we evaluate CoLoR-Filter empirically
  on two language modeling tasks: (1) selecting data from C4 for domain
  adaptation to evaluation on Books and (2) selecting data from C4 for a suite of
  downstream multiple-choice question answering tasks. We demonstrate favorable
  scaling both as we subselect more aggressively and using small auxiliary models
  to select data for large target models. As one headline result, CoLoR-Filter
  data selected using a pair of 150m parameter auxiliary models can train a 1.2b
  parameter target model to match a 1.2b parameter model trained on 25b randomly
  selected tokens with 25x less data for Books and 11x less data for the
  downstream tasks.
    Code: https://github.com/davidbrandfonbrener/color-filter-olmo
    Filtered data:
  https://huggingface.co/datasets/davidbrandfonbrener/color-filtered-c4
paper_ai_summary: A method called CoLoR-Filter uses an empirical Bayes-inspired approach to efficiently select high-quality data for training language models, demonstrating significant reductions in required data compared to random selection.
paper_ai_keywords: CoLoR-Filter Conditional Loss Reduction Filtering empirical Bayes-inspired approach language models data selection domain adaptation multiple-choice question answering parameter-efficient fine-tuning