title: Large-Scale Data Selection for Instruction Tuning
summary: Selecting high-quality training data from a larger pool is a crucial step
  when instruction-tuning language models, as carefully curated datasets often
  produce models that outperform those trained on much larger, noisier datasets.
  Automated data selection approaches for instruction-tuning are typically tested
  by selecting small datasets (roughly 10k samples) from small pools (100-200k
  samples). However, popular deployed instruction-tuned models often train on
  hundreds of thousands to millions of samples, subsampled from even larger data
  pools. We present a systematic study of how well data selection methods scale
  to these settings, selecting up to 2.5M samples from pools of up to 5.8M
  samples and evaluating across 7 diverse tasks. We show that many recently
  proposed methods fall short of random selection in this setting (while using
  more compute), and even decline in performance when given access to larger
  pools of data to select over. However, we find that a variant of
  representation-based data selection (RDS+), which uses weighted mean pooling of
  pretrained LM hidden states, consistently outperforms more complex methods
  across all settings tested -- all whilst being more compute-efficient. Our
  findings highlight that the scaling properties of proposed automated selection
  methods should be more closely examined. We release our code, data, and models
  at https://github.com/hamishivi/automated-instruction-selection.
paper_authors._id: 67c67ff6dec55d10cb10fc9e
paper_authors.name: Hamish Ivison
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-04T08:40:13.649Z
paper_authors._id: 62608fc2ffe8827cb1d89f9f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1654027835241-62608fc2ffe8827cb1d89f9f.png
paper_authors.fullname: Hamish Ivison
paper_authors.type: user
paper_authors.user: hamishivi
paper_authors._id: 67c67ff6dec55d10cb10fc9f
paper_authors.name: Muru Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-04T11:14:59.402Z
paper_authors._id: 61cc2cf4dcb47bd5ed3cd3b8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61cc2cf4dcb47bd5ed3cd3b8/O-tDNhSPZCZQSQG_BaEG9.png
paper_authors.fullname: Muru Zhang
paper_authors.type: user
paper_authors.user: nanami
paper_authors._id: 67c67ff6dec55d10cb10fca0
paper_authors.name: Faeze Brahman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-04T11:15:05.562Z
paper_authors._id: 65282b8d578679aac7888aec
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65282b8d578679aac7888aec/dibBkhH-z1c70mJZZxJ7u.jpeg
paper_authors.fullname: Faeze Brahman
paper_authors.type: user
paper_authors.user: faezeb
paper_authors._id: 67c67ff6dec55d10cb10fca1
paper_authors.name: Pang Wei Koh
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-04T11:15:14.558Z
paper_authors._id: 641b4263abfce26bcf7b27de
paper_authors.avatarUrl: /avatars/e91b4205e4f74b0dd8c333c23203a924.svg
paper_authors.fullname: Pang Wei Koh
paper_authors.type: user
paper_authors.user: pangwei
paper_authors._id: 67c67ff6dec55d10cb10fca2
paper_authors.name: Pradeep Dasigi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-04T11:15:20.400Z
paper_authors._id: 6408fcc93461c51cf735a61e
paper_authors.avatarUrl: /avatars/619f3653911d111f046a5a6c30fc8319.svg
paper_authors.fullname: Pradeep Dasigi
paper_authors.type: user
paper_authors.user: pradeepd
paper_title: Large-Scale Data Selection for Instruction Tuning
paper_summary: Selecting high-quality training data from a larger pool is a crucial step
  when instruction-tuning language models, as carefully curated datasets often
  produce models that outperform those trained on much larger, noisier datasets.
  Automated data selection approaches for instruction-tuning are typically tested
  by selecting small datasets (roughly 10k samples) from small pools (100-200k
  samples). However, popular deployed instruction-tuned models often train on
  hundreds of thousands to millions of samples, subsampled from even larger data
  pools. We present a systematic study of how well data selection methods scale
  to these settings, selecting up to 2.5M samples from pools of up to 5.8M
  samples and evaluating across 7 diverse tasks. We show that many recently
  proposed methods fall short of random selection in this setting (while using
  more compute), and even decline in performance when given access to larger
  pools of data to select over. However, we find that a variant of
  representation-based data selection (RDS+), which uses weighted mean pooling of
  pretrained LM hidden states, consistently outperforms more complex methods
  across all settings tested -- all whilst being more compute-efficient. Our
  findings highlight that the scaling properties of proposed automated selection
  methods should be more closely examined. We release our code, data, and models
  at https://github.com/hamishivi/automated-instruction-selection.
paper_ai_summary: A study on the scalability of data selection methods in instruction-tuning large language models reveals that a simpler, compute-efficient variant of representation-based data selection outperforms more complex methods.
paper_ai_keywords: instruction-tuning language models automated data selection representation-based data selection pretrained LM hidden states weighted mean pooling compute-efficiency