In-context Autoencoder for Context Compression in a Large Language Model
We propose the In-context Autoencoder (ICAE) for context compression in a
large language model (LLM). The ICAE has two modules: a learnable encoder
adapted with LoRA from an LLM for compressing a long context into a limited
number of memory slots, and a fixed decoder which is the target LLM that can
condition on the memory slots for various purposes. We first pretrain the ICAE
using both autoencoding and language modeling objectives on massive text data,
enabling it to generate memory slots that accurately and comprehensively
represent the original context. Then, we fine-tune the pretrained ICAE on a
small amount of instruct data to enhance its interaction with various prompts
for producing desirable responses. Our experimental results demonstrate that
the ICAE learned with our proposed pretraining and fine-tuning paradigm can
effectively produce memory slots with 4times context compression, which can
be well conditioned on by the target LLM to respond to various prompts. The
promising results demonstrate significant implications of the ICAE for its
novel approach to the long context problem and its potential to reduce
computation and memory overheads for LLM inference in practice, suggesting
further research effort in context management for an LLM. Our code and data
will be released shortly.
64b0a451dcc37b60d1d43997
Tao Ge
claimed_verified
2024-02-05T10:50:48.271Z
64ab5db78a1a9187c215f45a
/avatars/17add4cd02e183f9b9d4feb7b6c27c21.svg
Tao Ge
user
sggetao
64b0a451dcc37b60d1d43998
Jing Hu
64b0a451dcc37b60d1d43999
Xun Wang
64b0a451dcc37b60d1d4399a
Si-Qing Chen
64b0a451dcc37b60d1d4399b
Furu Wei
admin_assigned
2023-07-17T10:09:04.932Z
6368c512fbfe97c16a40baba
/avatars/1c23bc7c0b6d9225699ce27647623d7a.svg
Furu Wei
user
thegenerality
In-context Autoencoder for Context Compression in a Large Language Model
We propose the In-context Autoencoder (ICAE) for context compression in a
large language model (LLM). The ICAE has two modules: a learnable encoder
adapted with LoRA from an LLM for compressing a long context into a limited
number of memory slots, and a fixed decoder which is the target LLM that can
condition on the memory slots for various purposes. We first pretrain the ICAE
using both autoencoding and language modeling objectives on massive text data,
enabling it to generate memory slots that accurately and comprehensively
represent the original context. Then, we fine-tune the pretrained ICAE on a
small amount of instruct data to enhance its interaction with various prompts
for producing desirable responses. Our experimental results demonstrate that
the ICAE learned with our proposed pretraining and fine-tuning paradigm can
effectively produce memory slots with 4times context compression, which can
be well conditioned on by the target LLM to respond to various prompts. The
promising results demonstrate significant implications of the ICAE for its
novel approach to the long context problem and its potential to reduce
computation and memory overheads for LLM inference in practice, suggesting
further research effort in context management for an LLM. Our code and data
will be released shortly.
The In-context Autoencoder (ICAE) compresses long contexts into memory slots for efficient LLM inference using a combination of pretraining and fine-tuning.
In-context Autoencoder ICAE learnable encoder LoRA LLM memory slots autoencoding language modeling context compression pretraining fine-tuning instruct data prompts desirable responses context management