title: Thus Spake Long-Context Large Language Model
summary: Long context is an important topic in Natural Language Processing (NLP),
  running through the development of NLP architectures, and offers immense
  opportunities for Large Language Models (LLMs) giving LLMs the lifelong
  learning potential akin to humans. Unfortunately, the pursuit of a long context
  is accompanied by numerous obstacles. Nevertheless, long context remains a core
  competitive advantage for LLMs. In the past two years, the context length of
  LLMs has achieved a breakthrough extension to millions of tokens. Moreover, the
  research on long-context LLMs has expanded from length extrapolation to a
  comprehensive focus on architecture, infrastructure, training, and evaluation
  technologies.
    Inspired by the symphonic poem, Thus Spake Zarathustra, we draw an analogy
  between the journey of extending the context of LLM and the attempts of humans
  to transcend its mortality. In this survey, We will illustrate how LLM
  struggles between the tremendous need for a longer context and its equal need
  to accept the fact that it is ultimately finite. To achieve this, we give a
  global picture of the lifecycle of long-context LLMs from four perspectives:
  architecture, infrastructure, training, and evaluation, showcasing the full
  spectrum of long-context technologies. At the end of this survey, we will
  present 10 unanswered questions currently faced by long-context LLMs. We hope
  this survey can serve as a systematic introduction to the research on
  long-context LLMs.
paper_authors._id: 67bd37cb0d41e01cca99aa8b
paper_authors.name: Xiaoran Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-25T09:40:07.298Z
paper_authors._id: 64f033ef82c6eea604c4da8b
paper_authors.avatarUrl: /avatars/51b93fea7fd68b4274ee03701245dcca.svg
paper_authors.fullname: Liu Xiaoran
paper_authors.type: user
paper_authors.user: LiuXR
paper_authors._id: 67bd37cb0d41e01cca99aa8c
paper_authors.name: Ruixiao Li
paper_authors._id: 67bd37cb0d41e01cca99aa8d
paper_authors.name: Mianqiu Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-26T08:37:59.249Z
paper_authors._id: 6459c7c10aba070266e41bb1
paper_authors.avatarUrl: /avatars/2178cac69cf4123db5e85191160f3795.svg
paper_authors.fullname: mqhuang
paper_authors.type: user
paper_authors.user: LutherXD
paper_authors._id: 67bd37cb0d41e01cca99aa8e
paper_authors.name: Zhigeng Liu
paper_authors._id: 67bd37cb0d41e01cca99aa8f
paper_authors.name: Yuerong Song
paper_authors._id: 67bd37cb0d41e01cca99aa90
paper_authors.name: Qipeng Guo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-25T15:15:13.798Z
paper_authors._id: 6491cd52b1e5d3444528edb1
paper_authors.avatarUrl: /avatars/a85635d886c7f157b6723dec5c01c030.svg
paper_authors.fullname: Qipeng Guo
paper_authors.type: user
paper_authors.user: QipengGuo
paper_authors._id: 67bd37cb0d41e01cca99aa91
paper_authors.name: Siyang He
paper_authors._id: 67bd37cb0d41e01cca99aa92
paper_authors.name: Qiqi Wang
paper_authors._id: 67bd37cb0d41e01cca99aa93
paper_authors.name: Linlin Li
paper_authors._id: 67bd37cb0d41e01cca99aa94
paper_authors.name: Qun Liu
paper_authors._id: 67bd37cb0d41e01cca99aa95
paper_authors.name: Yaqian Zhou
paper_authors._id: 67bd37cb0d41e01cca99aa96
paper_authors.name: Xuanjing Huang
paper_authors._id: 67bd37cb0d41e01cca99aa97
paper_authors.name: Xipeng Qiu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-25T15:12:06.360Z
paper_authors._id: 61457b8deff2c9fdb4de4988
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1632381702899-61457b8deff2c9fdb4de4988.jpeg
paper_authors.fullname: Xipeng Qiu
paper_authors.type: user
paper_authors.user: xpqiu
paper_title: Thus Spake Long-Context Large Language Model
paper_summary: Long context is an important topic in Natural Language Processing (NLP),
  running through the development of NLP architectures, and offers immense
  opportunities for Large Language Models (LLMs) giving LLMs the lifelong
  learning potential akin to humans. Unfortunately, the pursuit of a long context
  is accompanied by numerous obstacles. Nevertheless, long context remains a core
  competitive advantage for LLMs. In the past two years, the context length of
  LLMs has achieved a breakthrough extension to millions of tokens. Moreover, the
  research on long-context LLMs has expanded from length extrapolation to a
  comprehensive focus on architecture, infrastructure, training, and evaluation
  technologies.
    Inspired by the symphonic poem, Thus Spake Zarathustra, we draw an analogy
  between the journey of extending the context of LLM and the attempts of humans
  to transcend its mortality. In this survey, We will illustrate how LLM
  struggles between the tremendous need for a longer context and its equal need
  to accept the fact that it is ultimately finite. To achieve this, we give a
  global picture of the lifecycle of long-context LLMs from four perspectives:
  architecture, infrastructure, training, and evaluation, showcasing the full
  spectrum of long-context technologies. At the end of this survey, we will
  present 10 unanswered questions currently faced by long-context LLMs. We hope
  this survey can serve as a systematic introduction to the research on
  long-context LLMs.
paper_ai_keywords: Long Language Models (LLMs) long context context length length extrapolation architecture infrastructure training evaluation technologies long-context LLMs long-context technologies