raw web page data, metadata extracts, and text extracts.

(2023) examines three data quality estimators, per-
plexity, Error L2-Norm (EL2N), and memorization
factor through data pruning, and surprisingly find
that pruning dataset based on perplexity performs
far better than more complicated techniques like
memorization.
DataManagementForLargeLanguageModels-ASurvey