title: ZIP-FIT: Embedding-Free Data Selection via Compression-Based Alignment
summary: Data selection is crucial for optimizing language model (LM) performance on
  specific tasks, yet most existing methods fail to effectively consider the
  target task distribution.
    Current approaches either ignore task-specific requirements entirely or rely
  on approximations that fail to capture the nuanced patterns needed for tasks
  like Autoformalization or code generation.
    Methods that do consider the target distribution often rely on simplistic,
  sometimes noisy, representations, like hashed n-gram features, which can lead
  to collisions and introduce noise.
    We introduce ZIP-FIT, a data selection framework that uses gzip compression
  to directly measure alignment between potential training data and the target
  task distribution.
    In extensive evaluations on Autoformalization and Python code generation,
  ZIP-FIT significantly outperforms leading baselines like DSIR and D4.
    Models trained on ZIP-FIT-selected data achieve their lowest cross-entropy
  loss up to 85.1\% faster than baselines, demonstrating that better task
  alignment leads to more efficient learning.
    In addition, ZIP-FIT performs selection up to 65.8\% faster than DSIR and two
  orders of magnitude faster than D4.
    Notably, ZIP-FIT shows that smaller, well-aligned datasets often outperform
  larger but less targeted ones, demonstrating that a small amount of higher
  quality data is superior to a large amount of lower quality data.
    Our results imply that task-aware data selection is crucial for efficient
  domain adaptation, and that compression offers a principled way to measure task
  alignment.
    By showing that targeted data selection can dramatically improve
  task-specific performance, our work provides new insights into the relationship
  between data quality, task alignment, and model learning efficiency.
paper_authors._id: 671c2017b91f5a4a2468bd4d
paper_authors.name: Elyas Obbad
paper_authors._id: 671c2017b91f5a4a2468bd4e
paper_authors.name: Iddah Mlauzi
paper_authors._id: 671c2017b91f5a4a2468bd4f
paper_authors.name: Brando Miranda
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-28T10:37:15.108Z
paper_authors._id: 60ec6453046c81764c7e06f0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60ec6453046c81764c7e06f0/gnKlFE5bkpvbDQNb2jrnV.png
paper_authors.fullname: Brando Miranda
paper_authors.type: user
paper_authors.user: brando
paper_authors._id: 671c2017b91f5a4a2468bd50
paper_authors.name: Rylan Schaeffer
paper_authors._id: 671c2017b91f5a4a2468bd51
paper_authors.name: Kamal Obbad
paper_authors._id: 671c2017b91f5a4a2468bd52
paper_authors.name: Suhana Bedi
paper_authors._id: 671c2017b91f5a4a2468bd53
paper_authors.name: Sanmi Koyejo
paper_title: ZIP-FIT: Embedding-Free Data Selection via Compression-Based Alignment
paper_summary: Data selection is crucial for optimizing language model (LM) performance on
  specific tasks, yet most existing methods fail to effectively consider the
  target task distribution.
    Current approaches either ignore task-specific requirements entirely or rely
  on approximations that fail to capture the nuanced patterns needed for tasks
  like Autoformalization or code generation.
    Methods that do consider the target distribution often rely on simplistic,
  sometimes noisy, representations, like hashed n-gram features, which can lead
  to collisions and introduce noise.
    We introduce ZIP-FIT, a data selection framework that uses gzip compression
  to directly measure alignment between potential training data and the target
  task distribution.
    In extensive evaluations on Autoformalization and Python code generation,
  ZIP-FIT significantly outperforms leading baselines like DSIR and D4.
    Models trained on ZIP-FIT-selected data achieve their lowest cross-entropy
  loss up to 85.1\% faster than baselines, demonstrating that better task
  alignment leads to more efficient learning.
    In addition, ZIP-FIT performs selection up to 65.8\% faster than DSIR and two
  orders of magnitude faster than D4.
    Notably, ZIP-FIT shows that smaller, well-aligned datasets often outperform
  larger but less targeted ones, demonstrating that a small amount of higher
  quality data is superior to a large amount of lower quality data.
    Our results imply that task-aware data selection is crucial for efficient
  domain adaptation, and that compression offers a principled way to measure task
  alignment.
    By showing that targeted data selection can dramatically improve
  task-specific performance, our work provides new insights into the relationship
  between data quality, task alignment, and model learning efficiency.
paper_ai_keywords: ZIP-FIT gzip compression task-specific requirements target task distribution uncertainty measurement Autoformalization code generation hashed n-gram features cross-entropy loss domain adaptation task alignment model learning efficiency