title: Instruction Mining: High-Quality Instruction Data Selection for Large
    Language Models
summary: Large language models typically undergo two training stages, pretraining and
  finetuning. Despite that large-scale pretraining endows the model with strong
  capabilities to generate natural language responses, these pretrained models
  can still fail to understand human instructions at times. To enhance language
  models' ability of interpreting and responding to instructions, instruction
  finetuning has emerged as a critical method in this area. Recent studies found
  that large language models can be finetuned to perform well even with a small
  amount of high-quality instruction-following data. However, the selection of
  high-quality datasets for finetuning language models still lacks clear
  guidelines to follow. In this paper, we propose InstructMining, a linear rule
  for evaluating instruction-following data quality. We formulate InstructMining
  using specific natural language indicators. To investigate the relationship
  between data quality and these indicators, we further conduct extensive
  finetuning experiments. The experiment results are then applied to estimating
  parameters in InstructMining. To further investigate its performance, we use
  InstructMining to select high-quality data from unseen datasets. Results
  demonstrate that InstructMining can help select relatively high-quality samples
  from various instruction-following datasets. Compared to models finetuned on
  unfiltered datasets, models finetuned on InstructMining selected datasets
  perform better on 42.5% cases.
paper_authors._id: 64af7e6a3b426e99c80e3be8
paper_authors.name: Yihan Cao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-17T06:48:34.111Z
paper_authors._id: 64b0332e252ed4adb7832978
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64b0332e252ed4adb7832978/dNNWNNonYzes-u2BylPrJ.png
paper_authors.fullname: Yihan Cao
paper_authors.type: user
paper_authors.user: instructmining
paper_authors._id: 64af7e6a3b426e99c80e3be9
paper_authors.name: Yanbin Kang
paper_authors._id: 64af7e6a3b426e99c80e3bea
paper_authors.name: Lichao Sun
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-21T08:29:53.285Z
paper_authors._id: 65a52766215aabac489e3468
paper_authors.avatarUrl: /avatars/fe05e22cd7e12e961296426434e17c76.svg
paper_authors.fullname: Lichao Sun
paper_authors.type: user
paper_authors.user: sunlichao137
paper_title: Instruction Mining: High-Quality Instruction Data Selection for Large
    Language Models
paper_summary: Large language models typically undergo two training stages, pretraining and
  finetuning. Despite that large-scale pretraining endows the model with strong
  capabilities to generate natural language responses, these pretrained models
  can still fail to understand human instructions at times. To enhance language
  models' ability of interpreting and responding to instructions, instruction
  finetuning has emerged as a critical method in this area. Recent studies found
  that large language models can be finetuned to perform well even with a small
  amount of high-quality instruction-following data. However, the selection of
  high-quality datasets for finetuning language models still lacks clear
  guidelines to follow. In this paper, we propose InstructMining, a linear rule
  for evaluating instruction-following data quality. We formulate InstructMining
  using specific natural language indicators. To investigate the relationship
  between data quality and these indicators, we further conduct extensive
  finetuning experiments. The experiment results are then applied to estimating
  parameters in InstructMining. To further investigate its performance, we use
  InstructMining to select high-quality data from unseen datasets. Results
  demonstrate that InstructMining can help select relatively high-quality samples
  from various instruction-following datasets. Compared to models finetuned on
  unfiltered datasets, models finetuned on InstructMining selected datasets
  perform better on 42.5% cases.
paper_ai_summary: InstructMining, a linear rule using natural language indicators, effectively selects high-quality instruction-following data for enhancing language model finetuning performance.
paper_ai_keywords: large language models pretraining finetuning instruction finetuning instruction-following data natural language indicators