Perplexed by Perplexity: Perplexity-Based Data Pruning With Small
  Reference Models
In this work, we investigate whether small language models can determine
high-quality subsets of large-scale text datasets that improve the performance
of larger language models. While existing work has shown that pruning based on
the perplexity of a larger model can yield high-quality data, we investigate
whether smaller models can be used for perplexity-based pruning and how pruning
is affected by the domain composition of the data being pruned. We demonstrate
that for multiple dataset compositions, perplexity-based pruning of pretraining
data can significantly improve downstream task performance: pruning
based on perplexities computed with a 125 million parameter model improves the
average performance on downstream tasks of a 3 billion parameter model by up to
2.04 and achieves up to a 1.45times reduction in pretraining steps to reach
commensurate baseline performance. Furthermore, we demonstrate that such
perplexity-based data pruning also yields downstream performance gains in the
over-trained and data-constrained regimes.
665d3d346c20caf090a2adee
Zachary Ankner
extracted_pending
2024-06-03T03:49:09.224Z
656fb8506871b53dba8d41f1
/avatars/6b84d97a32750b6082dacadd6efc0ad8.svg
Zachary Ankner
user
anknermit
665d3d346c20caf090a2adef
Cody Blakeney
admin_assigned
2024-06-03T08:55:29.398Z
642cc235b09c70b36de2be78
https://cdn-avatars.huggingface.co/v1/production/uploads/642cc235b09c70b36de2be78/K2P1S3Quw1btoJIhRs6j9.jpeg
Cody Blakeney
user
codestar12
665d3d346c20caf090a2adf0
Kartik Sreenivasan
admin_assigned
2024-06-03T08:55:36.535Z
648372e241796f46df60833c
/avatars/0ec44f753d4d2af73b1dafe3e324935a.svg
Kartik Sreenivasan
user
kartikmosaicml
665d3d346c20caf090a2adf1
Max Marion
admin_assigned
2024-06-03T08:55:51.946Z
654014e41389943fef371855
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/Gm9DLuhhaCTZhw31csePo.jpeg
Max Marion
user
maxisawesome
665d3d346c20caf090a2adf2
Matthew L. Leavitt
admin_assigned
2024-06-03T08:56:14.284Z
64937e776a38c3a61e858659
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/J19unBtIgCLdGQjkMujuZ.png
Matthew Leavitt
user
mleavitt
665d3d346c20caf090a2adf3
Mansheej Paul
admin_assigned
2024-06-03T08:56:20.482Z
622a8e4c9f6139daa8e99815
/avatars/d9beaabef72d89fe3c6a43937b78d977.svg
Mansheej Paul
user
mansheej
Perplexed by Perplexity: Perplexity-Based Data Pruning With Small
  Reference Models
In this work, we investigate whether small language models can determine
high-quality subsets of large-scale text datasets that improve the performance
of larger language models. While existing work has shown that pruning based on
the perplexity of a larger model can yield high-quality data, we investigate
whether smaller models can be used for perplexity-based pruning and how pruning
is affected by the domain composition of the data being pruned. We demonstrate
that for multiple dataset compositions, perplexity-based pruning of pretraining
data can significantly improve downstream task performance: pruning
based on perplexities computed with a 125 million parameter model improves the
average performance on downstream tasks of a 3 billion parameter model by up to
2.04 and achieves up to a 1.45times reduction in pretraining steps to reach
commensurate baseline performance. Furthermore, we demonstrate that such
perplexity-based data pruning also yields downstream performance gains in the
over-trained and data-constrained regimes.
Perplexity-based data pruning using small language models improves performance and reduces training time for larger models across various datasets and conditions.
perplexity-based pruning pretraining data downstream task performance parameter-efficient fine-tuning