How to Train Data-Efficient LLMs
The training of large language models (LLMs) is expensive. In this paper, we
study data-efficient approaches for pre-training LLMs, i.e., techniques that
aim to optimize the Pareto frontier of model quality and training resource/data
consumption. We seek to understand the tradeoffs associated with data selection
routines based on (i) expensive-to-compute data-quality estimates, and (ii)
maximization of coverage and diversity-based measures in the feature space. Our
first technique, Ask-LLM, leverages the zero-shot reasoning capabilities of
instruction-tuned LLMs to directly assess the quality of a training example. To
target coverage, we propose Density sampling, which models the data
distribution to select a diverse sample. In our comparison of 19 samplers,
involving hundreds of evaluation tasks and pre-training runs, we find that
Ask-LLM and Density are the best methods in their respective categories.
Coverage sampling can recover the performance of the full data, while models
trained on Ask-LLM data consistently outperform full-data training -- even when
we reject 90% of the original dataset, while converging up to 70% faster.
65cedabea87214d107c4ea4e
Noveen Sachdeva
claimed_verified
2024-02-16T08:36:15.317Z
652e529acb01191873067b02
/avatars/a340d5faca6ad1d120e8a904380276e9.svg
Sachdeva
user
Noveen
65cedabea87214d107c4ea4f
Benjamin Coleman
65cedabea87214d107c4ea50
Wang-Cheng Kang
65cedabea87214d107c4ea51
Jianmo Ni
65cedabea87214d107c4ea52
Lichan Hong
65cedabea87214d107c4ea53
Ed H. Chi
65cedabea87214d107c4ea54
James Caverlee
65cedabea87214d107c4ea55
Julian McAuley
65cedabea87214d107c4ea56
Derek Zhiyuan Cheng
admin_assigned
2024-02-16T10:30:00.884Z
624cedaa43ce7877c5af2c4e
/avatars/86f13f8e082e5c0b8ea671c374e4d675.svg
Zhiyuan Cheng
user
willcee
How to Train Data-Efficient LLMs
The training of large language models (LLMs) is expensive. In this paper, we
study data-efficient approaches for pre-training LLMs, i.e., techniques that
aim to optimize the Pareto frontier of model quality and training resource/data
consumption. We seek to understand the tradeoffs associated with data selection
routines based on (i) expensive-to-compute data-quality estimates, and (ii)
maximization of coverage and diversity-based measures in the feature space. Our
first technique, Ask-LLM, leverages the zero-shot reasoning capabilities of
instruction-tuned LLMs to directly assess the quality of a training example. To
target coverage, we propose Density sampling, which models the data
distribution to select a diverse sample. In our comparison of 19 samplers,
involving hundreds of evaluation tasks and pre-training runs, we find that
Ask-LLM and Density are the best methods in their respective categories.
Coverage sampling can recover the performance of the full data, while models
trained on Ask-LLM data consistently outperform full-data training -- even when
we reject 90% of the original dataset, while converging up to 70% faster.
Data-efficient methods like Ask-LLM and Density sampling improve model quality and training efficiency in large language models by optimizing data selection and coverage.
large language models data-efficient pre-training model quality training resource data consumption data selection zero-shot reasoning instruction-tuned training example density sampling coverage sampling evaluation tasks pre-training runs