They first hypothesize that the inference loss of a
fine-tuned model on an evaluation set can serve as
a proxy for data quality. Then, they introduce a set
of natural language indicators to predict the infer-
ence loss without actually fine-tuning an LLM, i.e.
the number of tokens in tokenized inputs and out-
puts, reward score (Köpf et al., 2023), perplexity,
Measure of Textual Lexical Diversity (McCarthy
and Jarvis, 2010), distance to approximate i-th
nearest neighbors (Dong et al., 2011) in Sentence-
BERT (Reimers and Gurevych, 2019) embedding
space, and the naturalness, coherence, and under-
standability scores provided by UniEval dialogue
model (Zhong et al., 2022).

Instruction Diversity
DataManagementForLargeLanguageModels-ASurvey