title: MIG: Automatic Data Selection for Instruction Tuning by Maximizing
    Information Gain in Semantic Space
summary: Data quality and diversity are key to the construction of effective
  instruction-tuning datasets. % With the increasing availability of open-source
  instruction-tuning datasets, it is advantageous to automatically select
  high-quality and diverse subsets from a vast amount of data. % Existing methods
  typically prioritize instance quality and use heuristic rules to maintain
  diversity. % However, this absence of a comprehensive view of the entire
  collection often leads to suboptimal results. % Moreover, heuristic rules
  generally focus on distance or clustering within the embedding space, which
  fails to accurately capture the intent of complex instructions in the semantic
  space. % To bridge this gap, we propose a unified method for quantifying the
  information content of datasets. This method models the semantic space by
  constructing a label graph and quantifies diversity based on the distribution
  of information within the graph. % Based on such a measurement, we further
  introduce an efficient sampling method that selects data samples iteratively to
  Maximize the Information Gain (MIG) in semantic
  space. % Experiments on various datasets and base models demonstrate that MIG
  consistently outperforms state-of-the-art methods. % Notably, the model
  fine-tuned with 5\% Tulu3 data sampled by MIG achieves comparable performance
  to the official SFT model trained on the full dataset, with improvements of
  +5.73\% on AlpacaEval and +6.89\% on Wildbench.
paper_authors._id: 6805b38355d3c792e1a9d0dd
paper_authors.name: Yicheng Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-22T09:59:49.780Z
paper_authors._id: 6752dcb6ec430bad7a26c83e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6752dcb6ec430bad7a26c83e/F685Ij72H1gIve4s9pcup.jpeg
paper_authors.fullname: Chen Yicheng
paper_authors.type: user
paper_authors.user: yichengchen24
paper_authors._id: 6805b38355d3c792e1a9d0de
paper_authors.name: Yining Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-22T09:59:52.428Z
paper_authors._id: 649988726677f66c2b486392
paper_authors.avatarUrl: /avatars/b649a77370660e129726e29504daba34.svg
paper_authors.fullname: Yining Li
paper_authors.type: user
paper_authors.user: ly015
paper_authors._id: 6805b38355d3c792e1a9d0df
paper_authors.name: Kai Hu
paper_authors._id: 6805b38355d3c792e1a9d0e0
paper_authors.name: Zerun Ma
paper_authors._id: 6805b38355d3c792e1a9d0e1
paper_authors.name: Haochen Ye
paper_authors._id: 6805b38355d3c792e1a9d0e2
paper_authors.name: Kai Chen
paper_title: MIG: Automatic Data Selection for Instruction Tuning by Maximizing
    Information Gain in Semantic Space
paper_summary: Data quality and diversity are key to the construction of effective
  instruction-tuning datasets. % With the increasing availability of open-source
  instruction-tuning datasets, it is advantageous to automatically select
  high-quality and diverse subsets from a vast amount of data. % Existing methods
  typically prioritize instance quality and use heuristic rules to maintain
  diversity. % However, this absence of a comprehensive view of the entire
  collection often leads to suboptimal results. % Moreover, heuristic rules
  generally focus on distance or clustering within the embedding space, which
  fails to accurately capture the intent of complex instructions in the semantic
  space. % To bridge this gap, we propose a unified method for quantifying the
  information content of datasets. This method models the semantic space by
  constructing a label graph and quantifies diversity based on the distribution
  of information within the graph. % Based on such a measurement, we further
  introduce an efficient sampling method that selects data samples iteratively to
  Maximize the Information Gain (MIG) in semantic
  space. % Experiments on various datasets and base models demonstrate that MIG
  consistently outperforms state-of-the-art methods. % Notably, the model
  fine-tuned with 5\% Tulu3 data sampled by MIG achieves comparable performance
  to the official SFT model trained on the full dataset, with improvements of
  +5.73\% on AlpacaEval and +6.89\% on Wildbench.
paper_ai_summary: A unified method quantifies dataset information content using a label graph and an efficient MIG sampling technique to enhance diversity and quality in instruction-tuning datasets, outperforming existing methods.
paper_ai_keywords: label graph information content semantic space MIG (Maximize the Information Gain) sampling method instruction-tuning datasets dataset diversity dataset quality heuristic rules AlpacaEval Wildbench