2.2.5 Diversity & Age
There are also works focusing on other aspects
of data management in the pretraining stage of
LLMs. For example, Lee et al. (2023a) show that
the formal diversities of publicly available pretrain-
ing datasets are high when measured by the recently
proposed Task2Vec diversity coefficient (Miranda
et al., 2022). They also demonstrate that the co-
efficient is aligned with the intuitive properties of
diversity and suggest using it in building more di-
verse datasets. Maharana et al. (2023) proposes
a novel pruning method D2 Pruning to balance
data diversity and difficulty in data selection by
representing a dataset as an undirected graph with
difficulty scores and adopting forward and reverse
message passing strategy to select a coreset encap-
sulating both diverse and difficult regions of the
dataset space.
DataManagementForLargeLanguageModels-ASurvey