title: How to Train Data-Efficient LLMs
summary: The training of large language models (LLMs) is expensive. In this paper, we
  study data-efficient approaches for pre-training LLMs, i.e., techniques that
  aim to optimize the Pareto frontier of model quality and training resource/data
  consumption. We seek to understand the tradeoffs associated with data selection
  routines based on (i) expensive-to-compute data-quality estimates, and (ii)
  maximization of coverage and diversity-based measures in the feature space. Our
  first technique, Ask-LLM, leverages the zero-shot reasoning capabilities of
  instruction-tuned LLMs to directly assess the quality of a training example. To
  target coverage, we propose Density sampling, which models the data
  distribution to select a diverse sample. In our comparison of 19 samplers,
  involving hundreds of evaluation tasks and pre-training runs, we find that
  Ask-LLM and Density are the best methods in their respective categories.
  Coverage sampling can recover the performance of the full data, while models
  trained on Ask-LLM data consistently outperform full-data training -- even when
  we reject 90% of the original dataset, while converging up to 70% faster.
paper_authors._id: 65cedabea87214d107c4ea4e
paper_authors.name: Noveen Sachdeva
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-16T08:36:15.317Z
paper_authors._id: 652e529acb01191873067b02
paper_authors.avatarUrl: /avatars/a340d5faca6ad1d120e8a904380276e9.svg
paper_authors.fullname: Sachdeva
paper_authors.type: user
paper_authors.user: Noveen
paper_authors._id: 65cedabea87214d107c4ea4f
paper_authors.name: Benjamin Coleman
paper_authors._id: 65cedabea87214d107c4ea50
paper_authors.name: Wang-Cheng Kang
paper_authors._id: 65cedabea87214d107c4ea51
paper_authors.name: Jianmo Ni
paper_authors._id: 65cedabea87214d107c4ea52
paper_authors.name: Lichan Hong
paper_authors._id: 65cedabea87214d107c4ea53
paper_authors.name: Ed H. Chi
paper_authors._id: 65cedabea87214d107c4ea54
paper_authors.name: James Caverlee
paper_authors._id: 65cedabea87214d107c4ea55
paper_authors.name: Julian McAuley
paper_authors._id: 65cedabea87214d107c4ea56
paper_authors.name: Derek Zhiyuan Cheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-16T10:30:00.884Z
paper_authors._id: 624cedaa43ce7877c5af2c4e
paper_authors.avatarUrl: /avatars/86f13f8e082e5c0b8ea671c374e4d675.svg
paper_authors.fullname: Zhiyuan Cheng
paper_authors.type: user
paper_authors.user: willcee
paper_title: How to Train Data-Efficient LLMs
paper_summary: The training of large language models (LLMs) is expensive. In this paper, we
  study data-efficient approaches for pre-training LLMs, i.e., techniques that
  aim to optimize the Pareto frontier of model quality and training resource/data
  consumption. We seek to understand the tradeoffs associated with data selection
  routines based on (i) expensive-to-compute data-quality estimates, and (ii)
  maximization of coverage and diversity-based measures in the feature space. Our
  first technique, Ask-LLM, leverages the zero-shot reasoning capabilities of
  instruction-tuned LLMs to directly assess the quality of a training example. To
  target coverage, we propose Density sampling, which models the data
  distribution to select a diverse sample. In our comparison of 19 samplers,
  involving hundreds of evaluation tasks and pre-training runs, we find that
  Ask-LLM and Density are the best methods in their respective categories.
  Coverage sampling can recover the performance of the full data, while models
  trained on Ask-LLM data consistently outperform full-data training -- even when
  we reject 90% of the original dataset, while converging up to 70% faster.
paper_ai_summary: Data-efficient methods like Ask-LLM and Density sampling improve model quality and training efficiency in large language models by optimizing data selection and coverage.
paper_ai_keywords: large language models data-efficient pre-training model quality training resource data consumption data selection zero-shot reasoning instruction-tuned training example density sampling coverage sampling evaluation tasks pre-training runs