2.2 Data Quality
High-quality data is crucial in the training of ma-
chine learning tasks according to previous stud-
ies (Jain et al., 2020; Gupta et al., 2021). In the pre-
training of LLMs, quality assurance techniques are
also adopted and usually form a data management
pipeline (Rae et al., 2021; Nguyen et al., 2023; Tiru-
mala et al., 2023), including deduplication, quality
filtering, and toxicity filtering. Other aspects like
social bias, data diversity, and data age are also
interesting topics in the research community.
DataManagementForLargeLanguageModels-ASurvey