title: In-context Autoencoder for Context Compression in a Large Language Model
summary: We propose the In-context Autoencoder (ICAE) for context compression in a
  large language model (LLM). The ICAE has two modules: a learnable encoder
  adapted with LoRA from an LLM for compressing a long context into a limited
  number of memory slots, and a fixed decoder which is the target LLM that can
  condition on the memory slots for various purposes. We first pretrain the ICAE
  using both autoencoding and language modeling objectives on massive text data,
  enabling it to generate memory slots that accurately and comprehensively
  represent the original context. Then, we fine-tune the pretrained ICAE on a
  small amount of instruct data to enhance its interaction with various prompts
  for producing desirable responses. Our experimental results demonstrate that
  the ICAE learned with our proposed pretraining and fine-tuning paradigm can
  effectively produce memory slots with 4times context compression, which can
  be well conditioned on by the target LLM to respond to various prompts. The
  promising results demonstrate significant implications of the ICAE for its
  novel approach to the long context problem and its potential to reduce
  computation and memory overheads for LLM inference in practice, suggesting
  further research effort in context management for an LLM. Our code and data
  will be released shortly.
paper_authors._id: 64b0a451dcc37b60d1d43997
paper_authors.name: Tao Ge
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-05T10:50:48.271Z
paper_authors._id: 64ab5db78a1a9187c215f45a
paper_authors.avatarUrl: /avatars/17add4cd02e183f9b9d4feb7b6c27c21.svg
paper_authors.fullname: Tao Ge
paper_authors.type: user
paper_authors.user: sggetao
paper_authors._id: 64b0a451dcc37b60d1d43998
paper_authors.name: Jing Hu
paper_authors._id: 64b0a451dcc37b60d1d43999
paper_authors.name: Xun Wang
paper_authors._id: 64b0a451dcc37b60d1d4399a
paper_authors.name: Si-Qing Chen
paper_authors._id: 64b0a451dcc37b60d1d4399b
paper_authors.name: Furu Wei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-17T10:09:04.932Z
paper_authors._id: 6368c512fbfe97c16a40baba
paper_authors.avatarUrl: /avatars/1c23bc7c0b6d9225699ce27647623d7a.svg
paper_authors.fullname: Furu Wei
paper_authors.type: user
paper_authors.user: thegenerality
paper_title: In-context Autoencoder for Context Compression in a Large Language Model
paper_summary: We propose the In-context Autoencoder (ICAE) for context compression in a
  large language model (LLM). The ICAE has two modules: a learnable encoder
  adapted with LoRA from an LLM for compressing a long context into a limited
  number of memory slots, and a fixed decoder which is the target LLM that can
  condition on the memory slots for various purposes. We first pretrain the ICAE
  using both autoencoding and language modeling objectives on massive text data,
  enabling it to generate memory slots that accurately and comprehensively
  represent the original context. Then, we fine-tune the pretrained ICAE on a
  small amount of instruct data to enhance its interaction with various prompts
  for producing desirable responses. Our experimental results demonstrate that
  the ICAE learned with our proposed pretraining and fine-tuning paradigm can
  effectively produce memory slots with 4times context compression, which can
  be well conditioned on by the target LLM to respond to various prompts. The
  promising results demonstrate significant implications of the ICAE for its
  novel approach to the long context problem and its potential to reduce
  computation and memory overheads for LLM inference in practice, suggesting
  further research effort in context management for an LLM. Our code and data
  will be released shortly.
paper_ai_summary: The In-context Autoencoder (ICAE) compresses long contexts into memory slots for efficient LLM inference using a combination of pretraining and fine-tuning.
paper_ai_keywords: In-context Autoencoder ICAE learnable encoder LoRA LLM memory slots autoencoding language modeling context compression pretraining fine-tuning instruct data prompts desirable responses context management