title: Efficient Data Selection at Scale via Influence Distillation
summary: Effective data selection is critical for efficient training of modern Large
  Language Models (LLMs). This paper introduces Influence Distillation, a novel,
  mathematically-justified framework for data selection that employs second-order
  information to optimally weight training samples. By distilling each sample's
  influence on a target distribution, our method assigns model-specific weights
  that are used to select training data for LLM fine-tuning, guiding it toward
  strong performance on the target domain. We derive these optimal weights for
  both Gradient Descent and Adam optimizers. To ensure scalability and reduce
  computational cost, we propose a landmark-based approximation:
  influence is precisely computed for a small subset of "landmark" samples and
  then efficiently propagated to all other samples to determine their weights. We
  validate Influence Distillation by applying it to instruction tuning on the
  Tulu V2 dataset, targeting a range of tasks including GSM8k, SQuAD, and MMLU,
  across several models from the Llama and Qwen families. Experiments show that
  Influence Distillation matches or outperforms state-of-the-art performance
  while achieving up to 3.5times faster selection.
paper_authors._id: 68383ce6023315653bf9c3be
paper_authors.name: Mahdi Nikdan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-29T20:19:11.613Z
paper_authors._id: 6526b8ebba9a8279c139616b
paper_authors.avatarUrl: /avatars/09f6b677603a03be128996a0765233e6.svg
paper_authors.fullname: Mahdi Nikdan
paper_authors.type: user
paper_authors.user: mnikdan97
paper_authors._id: 68383ce6023315653bf9c3bf
paper_authors.name: Vincent Cohen-Addad
paper_authors._id: 68383ce6023315653bf9c3c0
paper_authors.name: Dan Alistarh
paper_authors._id: 68383ce6023315653bf9c3c1
paper_authors.name: Vahab Mirrokni
paper_title: Efficient Data Selection at Scale via Influence Distillation
paper_summary: Effective data selection is critical for efficient training of modern Large
  Language Models (LLMs). This paper introduces Influence Distillation, a novel,
  mathematically-justified framework for data selection that employs second-order
  information to optimally weight training samples. By distilling each sample's
  influence on a target distribution, our method assigns model-specific weights
  that are used to select training data for LLM fine-tuning, guiding it toward
  strong performance on the target domain. We derive these optimal weights for
  both Gradient Descent and Adam optimizers. To ensure scalability and reduce
  computational cost, we propose a landmark-based approximation:
  influence is precisely computed for a small subset of "landmark" samples and
  then efficiently propagated to all other samples to determine their weights. We
  validate Influence Distillation by applying it to instruction tuning on the
  Tulu V2 dataset, targeting a range of tasks including GSM8k, SQuAD, and MMLU,
  across several models from the Llama and Qwen families. Experiments show that
  Influence Distillation matches or outperforms state-of-the-art performance
  while achieving up to 3.5times faster selection.
paper_ai_summary: Influence Distillation, using second-order information, optimally selects training data for LLM fine-tuning with landmark-based approximation, achieving faster and competitive performance on various tasks.
paper_ai_keywords: Influence Distillation second-order information model-specific weights Gradient Descent Adam optimizers landmark-based approximation LLM fine-tuning instruction tuning Tulu V2 dataset GSM8k SQuAD MMLU Llama Qwen