Thus Spake Long-Context Large Language Model
Long context is an important topic in Natural Language Processing (NLP),
running through the development of NLP architectures, and offers immense
opportunities for Large Language Models (LLMs) giving LLMs the lifelong
learning potential akin to humans. Unfortunately, the pursuit of a long context
is accompanied by numerous obstacles. Nevertheless, long context remains a core
competitive advantage for LLMs. In the past two years, the context length of
LLMs has achieved a breakthrough extension to millions of tokens. Moreover, the
research on long-context LLMs has expanded from length extrapolation to a
comprehensive focus on architecture, infrastructure, training, and evaluation
technologies.
  Inspired by the symphonic poem, Thus Spake Zarathustra, we draw an analogy
between the journey of extending the context of LLM and the attempts of humans
to transcend its mortality. In this survey, We will illustrate how LLM
struggles between the tremendous need for a longer context and its equal need
to accept the fact that it is ultimately finite. To achieve this, we give a
global picture of the lifecycle of long-context LLMs from four perspectives:
architecture, infrastructure, training, and evaluation, showcasing the full
spectrum of long-context technologies. At the end of this survey, we will
present 10 unanswered questions currently faced by long-context LLMs. We hope
this survey can serve as a systematic introduction to the research on
long-context LLMs.
67bd37cb0d41e01cca99aa8b
Xiaoran Liu
claimed_verified
2025-02-25T09:40:07.298Z
64f033ef82c6eea604c4da8b
/avatars/51b93fea7fd68b4274ee03701245dcca.svg
Liu Xiaoran
user
LiuXR
67bd37cb0d41e01cca99aa8c
Ruixiao Li
67bd37cb0d41e01cca99aa8d
Mianqiu Huang
claimed_verified
2025-02-26T08:37:59.249Z
6459c7c10aba070266e41bb1
/avatars/2178cac69cf4123db5e85191160f3795.svg
mqhuang
user
LutherXD
67bd37cb0d41e01cca99aa8e
Zhigeng Liu
67bd37cb0d41e01cca99aa8f
Yuerong Song
67bd37cb0d41e01cca99aa90
Qipeng Guo
admin_assigned
2025-02-25T15:15:13.798Z
6491cd52b1e5d3444528edb1
/avatars/a85635d886c7f157b6723dec5c01c030.svg
Qipeng Guo
user
QipengGuo
67bd37cb0d41e01cca99aa91
Siyang He
67bd37cb0d41e01cca99aa92
Qiqi Wang
67bd37cb0d41e01cca99aa93
Linlin Li
67bd37cb0d41e01cca99aa94
Qun Liu
67bd37cb0d41e01cca99aa95
Yaqian Zhou
67bd37cb0d41e01cca99aa96
Xuanjing Huang
67bd37cb0d41e01cca99aa97
Xipeng Qiu
admin_assigned
2025-02-25T15:12:06.360Z
61457b8deff2c9fdb4de4988
https://cdn-avatars.huggingface.co/v1/production/uploads/1632381702899-61457b8deff2c9fdb4de4988.jpeg
Xipeng Qiu
user
xpqiu
Thus Spake Long-Context Large Language Model
Long context is an important topic in Natural Language Processing (NLP),
running through the development of NLP architectures, and offers immense
opportunities for Large Language Models (LLMs) giving LLMs the lifelong
learning potential akin to humans. Unfortunately, the pursuit of a long context
is accompanied by numerous obstacles. Nevertheless, long context remains a core
competitive advantage for LLMs. In the past two years, the context length of
LLMs has achieved a breakthrough extension to millions of tokens. Moreover, the
research on long-context LLMs has expanded from length extrapolation to a
comprehensive focus on architecture, infrastructure, training, and evaluation
technologies.
  Inspired by the symphonic poem, Thus Spake Zarathustra, we draw an analogy
between the journey of extending the context of LLM and the attempts of humans
to transcend its mortality. In this survey, We will illustrate how LLM
struggles between the tremendous need for a longer context and its equal need
to accept the fact that it is ultimately finite. To achieve this, we give a
global picture of the lifecycle of long-context LLMs from four perspectives:
architecture, infrastructure, training, and evaluation, showcasing the full
spectrum of long-context technologies. At the end of this survey, we will
present 10 unanswered questions currently faced by long-context LLMs. We hope
this survey can serve as a systematic introduction to the research on
long-context LLMs.
The survey examines the advancements and challenges in long-context Large Language Models (LLMs), exploring the architecture, infrastructure, training, and evaluation technologies needed to extend their context length and address the inherent trade-offs.
Large Language Models long context length extrapolation architecture infrastructure training evaluation long-context LLMs