Extending Context Window of Large Language Models via Semantic
  Compression
Transformer-based Large Language Models (LLMs) often impose limitations on
the length of the text input to ensure the generation of fluent and relevant
responses. This constraint restricts their applicability in scenarios involving
long texts. We propose a novel semantic compression method that enables
generalization to texts that are 6-8 times longer, without incurring
significant computational costs or requiring fine-tuning. Our proposed
framework draws inspiration from source coding in information theory and
employs a pre-trained model to reduce the semantic redundancy of long inputs
before passing them to the LLMs for downstream tasks. Experimental results
demonstrate that our method effectively extends the context window of LLMs
across a range of tasks including question answering, summarization, few-shot
learning, and information retrieval. Furthermore, the proposed semantic
compression method exhibits consistent fluency in text generation while
reducing the associated computational overhead.
657fd75742fc53e18b88c436
Weizhi Fei
admin_assigned
2023-12-18T11:15:31.017Z
6430dcc08dee1e48c234e6e5
/avatars/8ffd87c55dd236c78c3154116c98e492.svg
Fei Weizhi
user
fwz
657fd75742fc53e18b88c437
Xueyan Niu
657fd75742fc53e18b88c438
Pingyi Zhou
admin_assigned
2023-12-18T11:15:57.124Z
5f55e3e98bf55658acfed1c8
/avatars/299d103ca32d62d11c7c800b74cd46e7.svg
zhoupingyi
user
zhoupingyi
657fd75742fc53e18b88c439
Lu Hou
admin_assigned
2023-12-18T11:16:03.722Z
5f86882fee5616341bc51da1
/avatars/b0c5883d6253ef67639d1cea591fa337.svg
Lu Hou
user
houlu369
657fd75742fc53e18b88c43a
Bo Bai
claimed_verified
2024-06-11T06:59:58.884Z
6667bdfd215b4c38f7ef17bb
/avatars/cad66dd576c3261ec5c276ae00e666d1.svg
Bo Bai
user
atomistic
657fd75742fc53e18b88c43b
Lei Deng
657fd75742fc53e18b88c43c
Wei Han
Extending Context Window of Large Language Models via Semantic
  Compression
Transformer-based Large Language Models (LLMs) often impose limitations on
the length of the text input to ensure the generation of fluent and relevant
responses. This constraint restricts their applicability in scenarios involving
long texts. We propose a novel semantic compression method that enables
generalization to texts that are 6-8 times longer, without incurring
significant computational costs or requiring fine-tuning. Our proposed
framework draws inspiration from source coding in information theory and
employs a pre-trained model to reduce the semantic redundancy of long inputs
before passing them to the LLMs for downstream tasks. Experimental results
demonstrate that our method effectively extends the context window of LLMs
across a range of tasks including question answering, summarization, few-shot
learning, and information retrieval. Furthermore, the proposed semantic
compression method exhibits consistent fluency in text generation while
reducing the associated computational overhead.
A semantic compression method extends the context window of Transformer-based LLMs for longer texts without fine-tuning, maintaining fluency and reducing computational costs.
Transformer-based LLMs semantic compression source coding information theory pre-trained model semantic redundancy context window question answering summarization few-shot learning information retrieval