title: Evaluating Sample Utility for Data Selection by Mimicking Model Weights
summary: Foundation models rely on large-scale web-crawled datasets, which frequently
  contain noisy data, biases, and irrelevant content. Existing data selection
  techniques typically use human heuristics, downstream evaluation datasets, or
  specialized scoring models, and can overlook samples' utility in the training
  process. Instead, we propose a new approach, Mimic Score, a data quality metric
  that uses a pretrained reference model as a guide to assess the usefulness of
  data samples for training a new model. It relies on the alignment between the
  gradient of the new model parameters and the vector pointing toward the
  reference model in weight space. Samples that misalign with this direction are
  considered low-value and can be filtered out. Motivated by the Mimic score, we
  develop Grad-Mimic, a data selection framework that identifies and prioritizes
  useful samples, automating the selection process to create effective filters.
  Empirically, using Mimic scores to guide model training results in consistent
  performance gains across six image datasets and enhances the performance of
  CLIP models. Moreover, Mimic scores and their associated filters improve upon
  existing filtering methods and offer accurate estimation of dataset quality.
paper_authors._id: 6786749192a268ed83174bfd
paper_authors.name: Tzu-Heng Huang
paper_authors._id: 6786749192a268ed83174bfe
paper_authors.name: Manjot Bilkhu
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-01-14T14:45:33.269Z
paper_authors._id: 659590c8994d0ef581417913
paper_authors.avatarUrl: /avatars/cef10191f915a99583a69e9e0ac8a330.svg
paper_authors.fullname: Manjot Bilkhu
paper_authors.type: user
paper_authors.user: mbilkhu
paper_authors._id: 6786749192a268ed83174bff
paper_authors.name: Frederic Sala
paper_authors._id: 6786749192a268ed83174c00
paper_authors.name: Javier Movellan
paper_title: Evaluating Sample Utility for Data Selection by Mimicking Model Weights
paper_summary: Foundation models rely on large-scale web-crawled datasets, which frequently
  contain noisy data, biases, and irrelevant content. Existing data selection
  techniques typically use human heuristics, downstream evaluation datasets, or
  specialized scoring models, and can overlook samples' utility in the training
  process. Instead, we propose a new approach, Mimic Score, a data quality metric
  that uses a pretrained reference model as a guide to assess the usefulness of
  data samples for training a new model. It relies on the alignment between the
  gradient of the new model parameters and the vector pointing toward the
  reference model in weight space. Samples that misalign with this direction are
  considered low-value and can be filtered out. Motivated by the Mimic score, we
  develop Grad-Mimic, a data selection framework that identifies and prioritizes
  useful samples, automating the selection process to create effective filters.
  Empirically, using Mimic scores to guide model training results in consistent
  performance gains across six image datasets and enhances the performance of
  CLIP models. Moreover, Mimic scores and their associated filters improve upon
  existing filtering methods and offer accurate estimation of dataset quality.
paper_ai_keywords: dense data biases irrelevant content data selection human heuristics downstream evaluation datasets specialized scoring models Mimic Score pretrained reference model data quality metric gradient weight space alignment Grad-Mimic data improvements useful samples CLIP models performance gains