An exactly solvable model for emergence and scaling laws
<|reference_start|>An exactly solvable model for emergence and scaling laws: Deep learning models can exhibit what appears to be a sudden ability to solve a new problem as training time, training data, or model size increases, a phenomenon known as emergence. In this paper, we present a framework where each new ability (a skill) is represented as a basis function. We solve a simple multi-linear model in this skill-basis, finding analytic expressions for the emergence of new skills, as well as for scaling laws of the loss with training time, data size, model size, and optimal compute ($C$). We compare our detailed calculations to direct simulations of a two-layer neural network trained on multitask sparse parity, where the tasks in the dataset are distributed according to a power-law. Our simple model captures, using a single fit parameter, the sigmoidal emergence of multiple new skills as training time, data size or model size increases in the neural network.<|reference_end|>
@article{nam2024an,
    title={An exactly solvable model for emergence and scaling laws in the
  multitask sparse parity problem},
    author={Yoonsoo Nam, Nayara Fonseca, Seok Hyeong Lee, Chris Mingard, Ard A.
  Louis},
    journal={arXiv preprint arXiv:2404.17563},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.17563},
    primaryClass={cs.LG cond-mat.dis-nn stat.ML}
}