Slaves to the Law of Large Numbers: An Asymptotic Equipartition Property for Perplexity in Generative Language Models
<|reference_start|>Slaves to the Law of Large Numbers: An Asymptotic Equipartition Property for Perplexity in Generative Language Models: We propose a new asymptotic equipartition property for the perplexity of a large piece of text generated by a language model and present theoretical arguments for this property. Perplexity, defined as a inverse likelihood function, is widely used as a performance metric for training language models. Our main result states that the logarithmic perplexity of any large text produced by a language model must asymptotically converge to the average entropy of its token distributions. This means that language models are constrained to only produce outputs from a ``typical set", which we show, is a vanishingly small subset of all possible grammatically correct outputs. We present preliminary experimental results from an open-source language model to support our theoretical claims. This work has possible practical applications for understanding and improving ``AI detection" tools and theoretical implications for the uniqueness, predictability and creative potential of generative models.<|reference_end|>
@article{mudumbai2024slaves,
    title={Slaves to the Law of Large Numbers: An Asymptotic Equipartition Property
  for Perplexity in Generative Language Models},
    author={Raghu Mudumbai and Tyler Bell},
    journal={arXiv preprint arXiv:2405.13798},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.13798},
    primaryClass={cs.CL cs.AI cs.IT math.IT}
}