title: BenTo: Benchmark Task Reduction with In-Context Transferability
summary: Evaluating large language models (LLMs) is costly: it requires the generation
  and examination of LLM outputs on a large-scale benchmark of various tasks.
  This paper investigates how to efficiently reduce the tasks used to benchmark
  LLMs without affecting the evaluation quality. Our study reveals that task
  transferability and relevance provide critical information to identify the most
  representative subset of tasks via optimizing a facility location function. We
  propose a practically efficient metric for estimating the transferability
  between two tasks via in-context learning (ICL). By analyzing the pairwise
  transferability, we can reduce tasks in a modern LLM benchmark (e.g., MMLU or
  FLAN) to 5% while inducing only a <4% difference to the evaluation on the
  original benchmark. Compared to prior works, our method is training-free,
  gradient-free, and highly efficient requiring ICL only.
paper_authors._id: 6711d1431133b5e4de7ef097
paper_authors.name: Hongyu Zhao
paper_authors._id: 6711d1431133b5e4de7ef098
paper_authors.name: Ming Li
paper_authors._id: 6711d1431133b5e4de7ef099
paper_authors.name: Lichao Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-18T13:00:45.492Z
paper_authors._id: 65a52766215aabac489e3468
paper_authors.avatarUrl: /avatars/fe05e22cd7e12e961296426434e17c76.svg
paper_authors.fullname: Lichao Sun
paper_authors.type: user
paper_authors.user: sunlichao137
paper_authors._id: 6711d1431133b5e4de7ef09a
paper_authors.name: Tianyi Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-18T08:12:25.345Z
paper_authors._id: 647f5af5b0e96764589f3b2a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/VJ4cDyjp5M3V5WmI5gPIU.jpeg
paper_authors.fullname: Tianyi Zhou
paper_authors.type: user
paper_authors.user: zhoutianyi
paper_title: BenTo: Benchmark Task Reduction with In-Context Transferability
paper_summary: Evaluating large language models (LLMs) is costly: it requires the generation
  and examination of LLM outputs on a large-scale benchmark of various tasks.
  This paper investigates how to efficiently reduce the tasks used to benchmark
  LLMs without affecting the evaluation quality. Our study reveals that task
  transferability and relevance provide critical information to identify the most
  representative subset of tasks via optimizing a facility location function. We
  propose a practically efficient metric for estimating the transferability
  between two tasks via in-context learning (ICL). By analyzing the pairwise
  transferability, we can reduce tasks in a modern LLM benchmark (e.g., MMLU or
  FLAN) to 5% while inducing only a <4% difference to the evaluation on the
  original benchmark. Compared to prior works, our method is training-free,
  gradient-free, and highly efficient requiring ICL only.
paper_ai_summary: Efficiently reducing the number of tasks for evaluating large language models with minimal impact on evaluation quality using a transferability metric based on in-context learning.
paper_ai_keywords: large language models evaluation facility location function transferability in-context learning