A Comprehensive Survey on Long Context Language Modeling
Efficient processing of long contexts has been a persistent pursuit in
Natural Language Processing. With the growing number of long documents,
dialogues, and other textual data, it is important to develop Long Context
Language Models (LCLMs) that can process and analyze extensive inputs in an
effective and efficient way. In this paper, we present a comprehensive survey
on recent advances in long-context modeling for large language models. Our
survey is structured around three key aspects: how to obtain effective and
efficient LCLMs, how to train and deploy LCLMs efficiently, and how to evaluate
and analyze LCLMs comprehensively. For the first aspect, we discuss data
strategies, architectural designs, and workflow approaches oriented with long
context processing. For the second aspect, we provide a detailed examination of
the infrastructure required for LCLM training and inference. For the third
aspect, we present evaluation paradigms for long-context comprehension and
long-form generation, as well as behavioral analysis and mechanism
interpretability of LCLMs. Beyond these three key aspects, we thoroughly
explore the diverse application scenarios where existing LCLMs have been
deployed and outline promising future development directions. This survey
provides an up-to-date review of the literature on long-context LLMs, which we
wish to serve as a valuable resource for both researchers and engineers. An
associated GitHub repository collecting the latest papers and repos is
available at:
https://github.com/LCLM-Horizon/A-Comprehensive-Survey-For-Long-Context-Language-Modeling{\color[RGB]{175,36,67}{LCLM-Horizon}}.
67e204089e7142f579abb84a
Jiaheng Liu
67e204089e7142f579abb84b
Dawei Zhu
claimed_verified
2025-03-25T08:20:47.031Z
64d2fce8129a210e569e0c76
/avatars/a79a832dc3a46ece1b9e542369fc4888.svg
Dawei Zhu
user
dwzhu
67e204089e7142f579abb84c
Zhiqi Bai
67e204089e7142f579abb84d
Yancheng He
67e204089e7142f579abb84e
Huanxuan Liao
claimed_verified
2025-03-25T08:20:43.983Z
634900f08146350b3a4b6710
https://cdn-avatars.huggingface.co/v1/production/uploads/1665728713794-noauth.jpeg
Xnhyacinth
user
Xnhyacinth
67e204089e7142f579abb84f
Haoran Que
67e204089e7142f579abb850
Zekun Wang
67e204089e7142f579abb851
Chenchen Zhang
67e204089e7142f579abb852
Ge Zhang
claimed_verified
2025-05-13T08:13:26.809Z
638efcf4c67af472d316d424
/avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
Ge Zhang
user
zhangysk
67e204089e7142f579abb853
Jiebin Zhang
67e204089e7142f579abb854
Yuanxing Zhang
67e204089e7142f579abb855
Zhuo Chen
67e204089e7142f579abb856
Hangyu Guo
67e204089e7142f579abb857
Shilong Li
67e204089e7142f579abb858
Ziqiang Liu
claimed_verified
2025-03-25T08:20:52.088Z
61b9b5f9fd429dff1ed3bd58
https://cdn-avatars.huggingface.co/v1/production/uploads/61b9b5f9fd429dff1ed3bd58/GrfeNU4K1zts-gmQdGbVE.jpeg
Ziqiang Liu
user
icoderzqliu
67e204089e7142f579abb859
Yong Shan
claimed_verified
2025-03-25T08:20:49.963Z
64b8cc7ebe76d2ff0703bfb3
/avatars/f1c7ff17fd923f1460d362333d9fbfe3.svg
sh
user
yo37
67e204089e7142f579abb85a
Yifan Song
67e204089e7142f579abb85b
Jiayi Tian
67e204089e7142f579abb85c
Wenhao Wu
67e204089e7142f579abb85d
Zhejian Zhou
claimed_verified
2025-03-25T08:20:54.197Z
61f650d9bf54777a520c599d
/avatars/9e7b4fc709835c0934bf2671298c9b33.svg
Jeremiah Zhou
user
JeremiahZ
67e204089e7142f579abb85e
Ruijie Zhu
claimed_verified
2025-03-25T08:20:56.102Z
63ff09f24852102d4871c19c
https://cdn-avatars.huggingface.co/v1/production/uploads/63ff09f24852102d4871c19c/lyE3xemtZss3qebK5sEXw.png
Rui-Jie Zhu
user
ridger
67e204089e7142f579abb85f
Junlan Feng
67e204089e7142f579abb860
Yang Gao
67e204089e7142f579abb861
Shizhu He
67e204089e7142f579abb862
Zhoujun Li
67e204089e7142f579abb863
Tianyu Liu
67e204089e7142f579abb864
Fanyu Meng
67e204089e7142f579abb865
Wenbo Su
67e204089e7142f579abb866
Yingshui Tan
67e204089e7142f579abb867
Zili Wang
67e204089e7142f579abb868
Jian Yang
67e204089e7142f579abb869
Wei Ye
67e204089e7142f579abb86a
Bo Zheng
67e204089e7142f579abb86b
Wangchunshu Zhou
claimed_verified
2025-06-18T12:17:57.731Z
628c8598ef14f971b698107f
/avatars/3a4ad87e6b5f9e836a1160d869df1447.svg
Zhou
user
Wangchunshu
67e204089e7142f579abb86c
Wenhao Huang
67e204089e7142f579abb86d
Sujian Li
67e204089e7142f579abb86e
Zhaoxiang Zhang
A Comprehensive Survey on Long Context Language Modeling
Efficient processing of long contexts has been a persistent pursuit in
Natural Language Processing. With the growing number of long documents,
dialogues, and other textual data, it is important to develop Long Context
Language Models (LCLMs) that can process and analyze extensive inputs in an
effective and efficient way. In this paper, we present a comprehensive survey
on recent advances in long-context modeling for large language models. Our
survey is structured around three key aspects: how to obtain effective and
efficient LCLMs, how to train and deploy LCLMs efficiently, and how to evaluate
and analyze LCLMs comprehensively. For the first aspect, we discuss data
strategies, architectural designs, and workflow approaches oriented with long
context processing. For the second aspect, we provide a detailed examination of
the infrastructure required for LCLM training and inference. For the third
aspect, we present evaluation paradigms for long-context comprehension and
long-form generation, as well as behavioral analysis and mechanism
interpretability of LCLMs. Beyond these three key aspects, we thoroughly
explore the diverse application scenarios where existing LCLMs have been
deployed and outline promising future development directions. This survey
provides an up-to-date review of the literature on long-context LLMs, which we
wish to serve as a valuable resource for both researchers and engineers. An
associated GitHub repository collecting the latest papers and repos is
available at:
https://github.com/LCLM-Horizon/A-Comprehensive-Survey-For-Long-Context-Language-Modeling{\color[RGB]{175,36,67}{LCLM-Horizon}}.
This survey examines recent advancements in long-context language modeling, focusing on methods for obtaining, training, deploying, and evaluating models capable of processing extensive textual inputs.
Long Context Language Models (LCLMs) long context processing data strategies architectural designs workflow approaches infrastructure long-context comprehension long-form generation behavioral analysis mechanism interpretability