Scaling Parameter-Constrained Language Models with Quality Data
<|reference_start|>Scaling Parameter-Constrained Language Models with Quality Data: Scaling laws in language modeling traditionally quantify training loss as a function of dataset size and model parameters, providing compute-optimal estimates but often neglecting the impact of data quality on model generalization. In this paper, we extend the conventional understanding of scaling law by offering a microscopic view of data quality within the original formulation -- effective training tokens -- which we posit to be a critical determinant of performance for parameter-constrained language models. Specifically, we formulate the proposed term of effective training tokens to be a combination of two readily-computed indicators of text: (i) text diversity and (ii) syntheticity as measured by a teacher model. We pretrained over $200$ models of 25M to 1.5B parameters on a diverse set of sampled, synthetic data, and estimated the constants that relate text quality, model size, training tokens, and eight reasoning task accuracy scores. We demonstrated the estimated constants yield +0.83 Pearson correlation with true accuracies, and analyzed it in scenarios involving widely-used data techniques such as data sampling and synthesis which aim to improve data quality.<|reference_end|>
@article{chang2024scaling,
    title={Scaling Parameter-Constrained Language Models with Quality Data},
    author={Ernie Chang, Matteo Paltenghi, Yang Li, Pin-Jie Lin, Changsheng Zhao,
  Patrick Huber, Zechun Liu, Rastislav Rabatin, Yangyang Shi, Vikas Chandra},
    journal={arXiv preprint arXiv:2410.03083},
    year={2024},
    archivePrefix={arXiv},
    eprint={2410.03083},
    primaryClass={cs.CL cs.AI}
}