An Information Theory of Compute-Optimal Size Scaling, Emergence, and Plateaus in Language Models
<|reference_start|>An Information Theory of Compute-Optimal Size Scaling, Emergence, and Plateaus in Language Models: Recent empirical studies show three phenomena with increasing size of language models: compute-optimal size scaling, emergent capabilities, and performance plateauing. We present a simple unified mathematical framework to explain all of these language model scaling phenomena, building on recent skill-text bipartite graph frameworks for semantic learning. Modeling the learning of concepts from texts as an iterative process yields an analogy to iterative decoding of low-density parity check (LDPC) codes in information theory. Thence, drawing on finite-size scaling characterizations of LDPC decoding, we derive the compute-optimal size scaling (Chinchilla rule) for language models. Further, using tools from random network theory, we provide a simple explanation for both emergence of complex skills and plateauing of performance as the size of language models scale. We see multiple plateaus.<|reference_end|>
@article{nayak2024an,
    title={An Information Theory of Compute-Optimal Size Scaling, Emergence, and
  Plateaus in Language Models},
    author={Anuj K. Nayak, Lav R. Varshney},
    journal={arXiv preprint arXiv:2410.01243},
    year={2024},
    archivePrefix={arXiv},
    eprint={2410.01243},
    primaryClass={cs.IT math.IT}
}