Evaluating Sample Utility for Data Selection by Mimicking Model Weights
Foundation models rely on large-scale web-crawled datasets, which frequently
contain noisy data, biases, and irrelevant content. Existing data selection
techniques typically use human heuristics, downstream evaluation datasets, or
specialized scoring models, and can overlook samples' utility in the training
process. Instead, we propose a new approach, Mimic Score, a data quality metric
that uses a pretrained reference model as a guide to assess the usefulness of
data samples for training a new model. It relies on the alignment between the
gradient of the new model parameters and the vector pointing toward the
reference model in weight space. Samples that misalign with this direction are
considered low-value and can be filtered out. Motivated by the Mimic score, we
develop Grad-Mimic, a data selection framework that identifies and prioritizes
useful samples, automating the selection process to create effective filters.
Empirically, using Mimic scores to guide model training results in consistent
performance gains across six image datasets and enhances the performance of
CLIP models. Moreover, Mimic scores and their associated filters improve upon
existing filtering methods and offer accurate estimation of dataset quality.
6786749192a268ed83174bfd
Tzu-Heng Huang
claimed_verified
2025-06-26T09:33:52.748Z
6363edf3de4bc2f294accb16
/avatars/9baddfec7170ec662e974116f561ab2c.svg
Tzu-Heng Huang
user
zihengh1
6786749192a268ed83174bfe
Manjot Bilkhu
extracted_confirmed
2025-01-14T14:45:33.269Z
659590c8994d0ef581417913
/avatars/cef10191f915a99583a69e9e0ac8a330.svg
Manjot Bilkhu
user
mbilkhu
6786749192a268ed83174bff
Frederic Sala
6786749192a268ed83174c00
Javier Movellan
Evaluating Sample Utility for Data Selection by Mimicking Model Weights
Foundation models rely on large-scale web-crawled datasets, which frequently
contain noisy data, biases, and irrelevant content. Existing data selection
techniques typically use human heuristics, downstream evaluation datasets, or
specialized scoring models, and can overlook samples' utility in the training
process. Instead, we propose a new approach, Mimic Score, a data quality metric
that uses a pretrained reference model as a guide to assess the usefulness of
data samples for training a new model. It relies on the alignment between the
gradient of the new model parameters and the vector pointing toward the
reference model in weight space. Samples that misalign with this direction are
considered low-value and can be filtered out. Motivated by the Mimic score, we
develop Grad-Mimic, a data selection framework that identifies and prioritizes
useful samples, automating the selection process to create effective filters.
Empirically, using Mimic scores to guide model training results in consistent
performance gains across six image datasets and enhances the performance of
CLIP models. Moreover, Mimic scores and their associated filters improve upon
existing filtering methods and offer accurate estimation of dataset quality.
Mimic Score and Grad-Mimic improve data selection by using gradient alignment to filter and prioritize useful samples, enhancing model training performance.
Mimic Score Grad-Mimic pretrained reference model weight space gradient alignment CLIP models dataset quality