Data Efficacy for Language Model Training
Data is fundamental to the training of language models (LM). Recent research
has been dedicated to data efficiency, which aims to maximize performance by
selecting a minimal or optimal subset of training data. Techniques such as data
filtering, sampling, and selection play a crucial role in this area. To
complement it, we define Data Efficacy, which focuses on maximizing performance
by optimizing the organization of training data and remains relatively
underexplored. This work introduces a general paradigm, DELT, for considering
data efficacy in LM training, which highlights the significance of training
data organization. DELT comprises three components: Data Scoring, Data
Selection, and Data Ordering. Among these components, we design
Learnability-Quality Scoring (LQS), as a new instance of Data Scoring, which
considers both the learnability and quality of each data sample from the
gradient consistency perspective. We also devise Folding Ordering (FO), as a
novel instance of Data Ordering, which addresses issues such as model
forgetting and data distribution bias. Comprehensive experiments validate the
data efficacy in LM training, which demonstrates the following: Firstly,
various instances of the proposed DELT enhance LM performance to varying
degrees without increasing the data scale and model size. Secondly, among these
instances, the combination of our proposed LQS for data scoring and Folding for
data ordering achieves the most significant improvement. Lastly, data efficacy
can be achieved together with data efficiency by applying data selection.
Therefore, we believe that data efficacy is a promising foundational area in LM
training.
6864cbbad59a9eda59024b40
Yalun Dai
6864cbbad59a9eda59024b41
Yangyu Huang
admin_assigned
2025-07-02T16:10:20.088Z
64c66647725ffa04b2fd6c94
/avatars/620f63f27fa1e90423b0dc22aa8e5809.svg
yangyu huang
user
yangyu90
6864cbbad59a9eda59024b42
Xin Zhang
6864cbbad59a9eda59024b43
Wenshan Wu
6864cbbad59a9eda59024b44
Chong Li
6864cbbad59a9eda59024b45
Wenhui Lu
6864cbbad59a9eda59024b46
Shijie Cao
6864cbbad59a9eda59024b47
Li Dong
6864cbbad59a9eda59024b48
Scarlett Li
admin_assigned
2025-07-02T16:10:29.486Z
685802f14008a9347510f1d2
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/8q7mZyRELYdK7BHYf1C9w.png
Scarlett Li
user
Scarlettli
Data Efficacy for Language Model Training
Data is fundamental to the training of language models (LM). Recent research
has been dedicated to data efficiency, which aims to maximize performance by
selecting a minimal or optimal subset of training data. Techniques such as data
filtering, sampling, and selection play a crucial role in this area. To
complement it, we define Data Efficacy, which focuses on maximizing performance
by optimizing the organization of training data and remains relatively
underexplored. This work introduces a general paradigm, DELT, for considering
data efficacy in LM training, which highlights the significance of training
data organization. DELT comprises three components: Data Scoring, Data
Selection, and Data Ordering. Among these components, we design
Learnability-Quality Scoring (LQS), as a new instance of Data Scoring, which
considers both the learnability and quality of each data sample from the
gradient consistency perspective. We also devise Folding Ordering (FO), as a
novel instance of Data Ordering, which addresses issues such as model
forgetting and data distribution bias. Comprehensive experiments validate the
data efficacy in LM training, which demonstrates the following: Firstly,
various instances of the proposed DELT enhance LM performance to varying
degrees without increasing the data scale and model size. Secondly, among these
instances, the combination of our proposed LQS for data scoring and Folding for
data ordering achieves the most significant improvement. Lastly, data efficacy
can be achieved together with data efficiency by applying data selection.
Therefore, we believe that data efficacy is a promising foundational area in LM
training.
DELT, a paradigm for optimizing data organization in language model training, enhances performance through Data Scoring, Data Selection, and Data Ordering, with Learnability-Quality Scoring and Folding Ordering as key components.
Data Efficacy DELT Data Scoring Data Selection Data Ordering Learnability-Quality Scoring Folding Ordering model forgetting data distribution bias