title: ICon: In-Context Contribution for Automatic Data Selection
summary: Data selection for instruction tuning is essential for improving the
  performance of Large Language Models (LLMs) and reducing training cost.
  However, existing automated selection methods either depend on computationally
  expensive gradient-based measures or manually designed heuristics, which may
  fail to fully exploit the intrinsic attributes of data. In this paper, we
  propose In-context Learning for Contribution Measurement (ICon), a novel
  gradient-free method that takes advantage of the implicit fine-tuning nature of
  in-context learning (ICL) to measure sample contribution without gradient
  computation or manual indicators engineering. ICon offers a computationally
  efficient alternative to gradient-based methods and reduces human inductive
  bias inherent in heuristic-based approaches. ICon comprises three components
  and identifies high-contribution data by assessing performance shifts under
  implicit learning through ICL. Extensive experiments on three LLMs across 12
  benchmarks and 5 pairwise evaluation sets demonstrate the effectiveness of
  ICon. Remarkably, on LLaMA3.1-8B, models trained on 15% of ICon-selected data
  outperform full datasets by 5.42% points and exceed the best performance of
  widely used selection methods by 2.06% points. We further analyze
  high-contribution samples selected by ICon, which show both diverse tasks and
  appropriate difficulty levels, rather than just the hardest ones.
paper_authors._id: 681d95bc11abe59dc97e4c5a
paper_authors.name: Yixin Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-09T07:21:18.753Z
paper_authors._id: 647e99d9becb41a272970ca4
paper_authors.avatarUrl: /avatars/291831643937a298c5903c6dc037b950.svg
paper_authors.fullname: Ann
paper_authors.type: user
paper_authors.user: yyxsghx
paper_authors._id: 681d95bc11abe59dc97e4c5b
paper_authors.name: Qingxiu Dong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-09T08:57:38.844Z
paper_authors._id: 670740744341dcee459fb990
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/66UkZvrAk7fQr5YCylEFk.png
paper_authors.fullname: Qingxiu Dong
paper_authors.type: user
paper_authors.user: Rsy24
paper_authors._id: 681d95bc11abe59dc97e4c5c
paper_authors.name: Linli Yao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-09T08:57:47.561Z
paper_authors._id: 655ca347f426a304c6b393a1
paper_authors.avatarUrl: /avatars/67f0310d59c5912d38c2ad8e6448614d.svg
paper_authors.fullname: Linli Yao
paper_authors.type: user
paper_authors.user: yaolily
paper_authors._id: 681d95bc11abe59dc97e4c5d
paper_authors.name: Fangwei Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-09T08:57:53.796Z
paper_authors._id: 654cca3fe1b4cd6d40d5a7ae
paper_authors.avatarUrl: /avatars/4d09531277e16ad71474fc888c16b227.svg
paper_authors.fullname: Fangwei Zhu
paper_authors.type: user
paper_authors.user: soliz1998
paper_authors._id: 681d95bc11abe59dc97e4c5e
paper_authors.name: Zhifang Sui
paper_title: ICon: In-Context Contribution for Automatic Data Selection
paper_summary: Data selection for instruction tuning is essential for improving the
  performance of Large Language Models (LLMs) and reducing training cost.
  However, existing automated selection methods either depend on computationally
  expensive gradient-based measures or manually designed heuristics, which may
  fail to fully exploit the intrinsic attributes of data. In this paper, we
  propose In-context Learning for Contribution Measurement (ICon), a novel
  gradient-free method that takes advantage of the implicit fine-tuning nature of
  in-context learning (ICL) to measure sample contribution without gradient
  computation or manual indicators engineering. ICon offers a computationally
  efficient alternative to gradient-based methods and reduces human inductive
  bias inherent in heuristic-based approaches. ICon comprises three components
  and identifies high-contribution data by assessing performance shifts under
  implicit learning through ICL. Extensive experiments on three LLMs across 12
  benchmarks and 5 pairwise evaluation sets demonstrate the effectiveness of
  ICon. Remarkably, on LLaMA3.1-8B, models trained on 15% of ICon-selected data
  outperform full datasets by 5.42% points and exceed the best performance of
  widely used selection methods by 2.06% points. We further analyze
  high-contribution samples selected by ICon, which show both diverse tasks and
  appropriate difficulty levels, rather than just the hardest ones.
paper_ai_summary: In-context Learning for Contribution Measurement (ICon) is a gradient-free method that identifies high-contribution data for instruction tuning of Large Language Models, outperforming gradient-based and heuristic-based selection methods.
paper_ai_keywords: in-context learning ICon Large Language Models LLMs gradient-free method sample contribution performance shifts implicit learning ICL