title: Perplexed by Perplexity: Perplexity-Based Data Pruning With Small
    Reference Models
summary: In this work, we investigate whether small language models can determine
  high-quality subsets of large-scale text datasets that improve the performance
  of larger language models. While existing work has shown that pruning based on
  the perplexity of a larger model can yield high-quality data, we investigate
  whether smaller models can be used for perplexity-based pruning and how pruning
  is affected by the domain composition of the data being pruned. We demonstrate
  that for multiple dataset compositions, perplexity-based pruning of pretraining
  data can significantly improve downstream task performance: pruning
  based on perplexities computed with a 125 million parameter model improves the
  average performance on downstream tasks of a 3 billion parameter model by up to
  2.04 and achieves up to a 1.45times reduction in pretraining steps to reach
  commensurate baseline performance. Furthermore, we demonstrate that such
  perplexity-based data pruning also yields downstream performance gains in the
  over-trained and data-constrained regimes.
paper_authors._id: 665d3d346c20caf090a2adee
paper_authors.name: Zachary Ankner
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-06-03T03:49:09.224Z
paper_authors._id: 656fb8506871b53dba8d41f1
paper_authors.avatarUrl: /avatars/6b84d97a32750b6082dacadd6efc0ad8.svg
paper_authors.fullname: Zachary Ankner
paper_authors.type: user
paper_authors.user: anknermit
paper_authors._id: 665d3d346c20caf090a2adef
paper_authors.name: Cody Blakeney
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-03T08:55:29.398Z
paper_authors._id: 642cc235b09c70b36de2be78
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/642cc235b09c70b36de2be78/K2P1S3Quw1btoJIhRs6j9.jpeg
paper_authors.fullname: Cody Blakeney
paper_authors.type: user
paper_authors.user: codestar12
paper_authors._id: 665d3d346c20caf090a2adf0
paper_authors.name: Kartik Sreenivasan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-03T08:55:36.535Z
paper_authors._id: 648372e241796f46df60833c
paper_authors.avatarUrl: /avatars/0ec44f753d4d2af73b1dafe3e324935a.svg
paper_authors.fullname: Kartik Sreenivasan
paper_authors.type: user
paper_authors.user: kartikmosaicml
paper_authors._id: 665d3d346c20caf090a2adf1
paper_authors.name: Max Marion
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-03T08:55:51.946Z
paper_authors._id: 654014e41389943fef371855
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/Gm9DLuhhaCTZhw31csePo.jpeg
paper_authors.fullname: Max Marion
paper_authors.type: user
paper_authors.user: maxisawesome
paper_authors._id: 665d3d346c20caf090a2adf2
paper_authors.name: Matthew L. Leavitt
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-03T08:56:14.284Z
paper_authors._id: 64937e776a38c3a61e858659
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/J19unBtIgCLdGQjkMujuZ.png
paper_authors.fullname: Matthew Leavitt
paper_authors.type: user
paper_authors.user: mleavitt
paper_authors._id: 665d3d346c20caf090a2adf3
paper_authors.name: Mansheej Paul
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-03T08:56:20.482Z
paper_authors._id: 622a8e4c9f6139daa8e99815
paper_authors.avatarUrl: /avatars/d9beaabef72d89fe3c6a43937b78d977.svg
paper_authors.fullname: Mansheej Paul
paper_authors.type: user
paper_authors.user: mansheej
paper_title: Perplexed by Perplexity: Perplexity-Based Data Pruning With Small
    Reference Models
paper_summary: In this work, we investigate whether small language models can determine
  high-quality subsets of large-scale text datasets that improve the performance
  of larger language models. While existing work has shown that pruning based on
  the perplexity of a larger model can yield high-quality data, we investigate
  whether smaller models can be used for perplexity-based pruning and how pruning
  is affected by the domain composition of the data being pruned. We demonstrate
  that for multiple dataset compositions, perplexity-based pruning of pretraining
  data can significantly improve downstream task performance: pruning
  based on perplexities computed with a 125 million parameter model improves the
  average performance on downstream tasks of a 3 billion parameter model by up to
  2.04 and achieves up to a 1.45times reduction in pretraining steps to reach
  commensurate baseline performance. Furthermore, we demonstrate that such
  perplexity-based data pruning also yields downstream performance gains in the
  over-trained and data-constrained regimes.
paper_ai_keywords: perplexity-based pruning pretraining data downstream tasks over-trained data-constrained