Beyond the Black Box: A Statistical Model for LLM Reasoning and Inference
<|reference_start|>Beyond the Black Box: A Statistical Model for LLM Reasoning and Inference: This paper introduces a novel Bayesian learning model to explain the behavior of Large Language Models (LLMs), focusing on their core optimization metric of next token prediction. We develop a theoretical framework based on an ideal generative text model represented by a multinomial transition probability matrix with a prior, and examine how LLMs approximate this matrix. Key contributions include: (i) a continuity theorem relating embeddings to multinomial distributions, (ii) a demonstration that LLM text generation aligns with Bayesian learning principles, (iii) an explanation for the emergence of in-context learning in larger models, (iv) empirical validation using visualizations of next token probabilities from an instrumented Llama model Our findings provide new insights into LLM functioning, offering a statistical foundation for understanding their capabilities and limitations. This framework has implications for LLM design, training, and application, potentially guiding future developments in the field.<|reference_end|>
@article{dalal2024beyond,
    title={Beyond the Black Box: A Statistical Model for LLM Reasoning and
  Inference},
    author={Siddhartha Dalal and Vishal Misra},
    journal={arXiv preprint arXiv:2402.03175},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.03175},
    primaryClass={cs.LG cs.AI}
}