5 Conclusions
This paper makes the first attempt to overview data
management in the training of LLMs. We discuss
the pretraining and supervised fine-tuning stage
of LLM successively and summarize the up-to-
date research efforts into data quantity, data qual-
ity, and domain/task composition for each stage.
Data management systems in the pretraining stage
and Data-efficient learning in the supervised fine-
tuning stage are also discussed. Finally, we high-
light several challenges and promising future direc-
tions for LLM training data management. We hope
this survey can provide insightful guidance for prac-
titioners and inspire further research in effective
and efficient data management for the development
of LLMs.
DataManagementForLargeLanguageModels-ASurvey