title: Extending Context Window of Large Language Models via Semantic
    Compression
summary: Transformer-based Large Language Models (LLMs) often impose limitations on
  the length of the text input to ensure the generation of fluent and relevant
  responses. This constraint restricts their applicability in scenarios involving
  long texts. We propose a novel semantic compression method that enables
  generalization to texts that are 6-8 times longer, without incurring
  significant computational costs or requiring fine-tuning. Our proposed
  framework draws inspiration from source coding in information theory and
  employs a pre-trained model to reduce the semantic redundancy of long inputs
  before passing them to the LLMs for downstream tasks. Experimental results
  demonstrate that our method effectively extends the context window of LLMs
  across a range of tasks including question answering, summarization, few-shot
  learning, and information retrieval. Furthermore, the proposed semantic
  compression method exhibits consistent fluency in text generation while
  reducing the associated computational overhead.
paper_authors._id: 657fd75742fc53e18b88c436
paper_authors.name: Weizhi Fei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-18T11:15:31.017Z
paper_authors._id: 6430dcc08dee1e48c234e6e5
paper_authors.avatarUrl: /avatars/8ffd87c55dd236c78c3154116c98e492.svg
paper_authors.fullname: Fei Weizhi
paper_authors.type: user
paper_authors.user: fwz
paper_authors._id: 657fd75742fc53e18b88c437
paper_authors.name: Xueyan Niu
paper_authors._id: 657fd75742fc53e18b88c438
paper_authors.name: Pingyi Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-18T11:15:57.124Z
paper_authors._id: 5f55e3e98bf55658acfed1c8
paper_authors.avatarUrl: /avatars/299d103ca32d62d11c7c800b74cd46e7.svg
paper_authors.fullname: zhoupingyi
paper_authors.type: user
paper_authors.user: zhoupingyi
paper_authors._id: 657fd75742fc53e18b88c439
paper_authors.name: Lu Hou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-18T11:16:03.722Z
paper_authors._id: 5f86882fee5616341bc51da1
paper_authors.avatarUrl: /avatars/b0c5883d6253ef67639d1cea591fa337.svg
paper_authors.fullname: Lu Hou
paper_authors.type: user
paper_authors.user: houlu369
paper_authors._id: 657fd75742fc53e18b88c43a
paper_authors.name: Bo Bai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-11T06:59:58.884Z
paper_authors._id: 6667bdfd215b4c38f7ef17bb
paper_authors.avatarUrl: /avatars/cad66dd576c3261ec5c276ae00e666d1.svg
paper_authors.fullname: Bo Bai
paper_authors.type: user
paper_authors.user: atomistic
paper_authors._id: 657fd75742fc53e18b88c43b
paper_authors.name: Lei Deng
paper_authors._id: 657fd75742fc53e18b88c43c
paper_authors.name: Wei Han
paper_title: Extending Context Window of Large Language Models via Semantic
    Compression
paper_summary: Transformer-based Large Language Models (LLMs) often impose limitations on
  the length of the text input to ensure the generation of fluent and relevant
  responses. This constraint restricts their applicability in scenarios involving
  long texts. We propose a novel semantic compression method that enables
  generalization to texts that are 6-8 times longer, without incurring
  significant computational costs or requiring fine-tuning. Our proposed
  framework draws inspiration from source coding in information theory and
  employs a pre-trained model to reduce the semantic redundancy of long inputs
  before passing them to the LLMs for downstream tasks. Experimental results
  demonstrate that our method effectively extends the context window of LLMs
  across a range of tasks including question answering, summarization, few-shot
  learning, and information retrieval. Furthermore, the proposed semantic
  compression method exhibits consistent fluency in text generation while
  reducing the associated computational overhead.
paper_ai_keywords: Transformer-based Large Language models (LLMs) semanticating context length limitations semantic compression prompt generation source coding information theory pre-trained model semantic redundancy context window question answering summarization few-shot learning information retrieval computational overhead