Improving Pretraining Data Using Perplexity Correlations
<|reference_start|>Improving Pretraining Data Using Perplexity Correlations: Quality pretraining data is often seen as the key to high-performance language models. However, progress in understanding pretraining data has been slow due to the costly pretraining runs required for data selection experiments. We present a framework that avoids these costs and selects high-quality pretraining data without any LLM training of our own. Our work is based on a simple observation: LLM losses on many pretraining texts are correlated with downstream benchmark performance, and selecting high-correlation documents is an effective pretraining data selection method. We build a new statistical framework for data selection centered around estimates of perplexity-benchmark correlations and perform data selection using a sample of 90 LLMs taken from the Open LLM Leaderboard on texts from tens of thousands of web domains. In controlled pretraining experiments at the 160M parameter scale on 8 benchmarks, our approach outperforms DSIR on every benchmark, while matching the best data selector found in DataComp-LM, a hand-engineered bigram classifier.<|reference_end|>
@article{thrush2024improving,
    title={Improving Pretraining Data Using Perplexity Correlations},
    author={Tristan Thrush, Christopher Potts, Tatsunori Hashimoto},
    journal={arXiv preprint arXiv:2409.05816},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.05816},
    primaryClass={cs.CL cs.LG stat.ML}
}