Critical Phase Transition in a Large Language Model
<|reference_start|>Critical Phase Transition in a Large Language Model: The performance of large language models (LLMs) strongly depends on the \textit{temperature} parameter. Empirically, at very low temperatures, LLMs generate sentences with clear repetitive structures, while at very high temperatures, generated sentences are often incomprehensible. In this study, using GPT-2, we numerically demonstrate that the difference between the two regimes is not just a smooth change but a phase transition with singular, divergent statistical quantities. Our extensive analysis shows that critical behaviors, such as a power-law decay of correlation in a text, emerge in the LLM at the transition temperature as well as in a natural language dataset. We also discuss that several statistical quantities characterizing the criticality should be useful to evaluate the performance of LLMs.<|reference_end|>
@article{nakaishi2024critical,
    title={Critical Phase Transition in Large Language Models},
    author={Kai Nakaishi, Yoshihiko Nishikawa, Koji Hukushima},
    journal={arXiv preprint arXiv:2406.05335},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.05335},
    primaryClass={cond-mat.dis-nn cs.LG}
}