title: LLoCO: Learning Long Contexts Offline
summary: Processing long contexts remains a challenge for large language models (LLMs)
  due to the quadratic computational and memory overhead of the self-attention
  mechanism and the substantial KV cache sizes during generation. We propose a
  novel approach to address this problem by learning contexts offline through
  context compression and in-domain parameter-efficient finetuning. Our method
  enables an LLM to create a concise representation of the original context and
  efficiently retrieve relevant information to answer questions accurately. We
  introduce LLoCO, a technique that combines context compression, retrieval, and
  parameter-efficient finetuning using LoRA. Our approach extends the effective
  context window of a 4k token LLaMA2-7B model to handle up to 128k tokens. We
  evaluate our approach on several long-context question-answering datasets,
  demonstrating that LLoCO significantly outperforms in-context learning while
  using 30times fewer tokens during inference. LLoCO achieves up to
  7.62times speed-up and substantially reduces the cost of long document
  question answering, making it a promising solution for efficient long context
  processing. Our code is publicly available at
  https://github.com/jeffreysijuntan/lloco.
paper_authors._id: 661896d0de161bdcb0f59be4
paper_authors.name: Sijun Tan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T12:30:36.653Z
paper_authors._id: 652867b09903f7a1c9f7cbf1
paper_authors.avatarUrl: /avatars/1d1d3665328943ccdbd6a25ce03f2f10.svg
paper_authors.fullname: Sijun Tan
paper_authors.type: user
paper_authors.user: sijuntan
paper_authors._id: 661896d0de161bdcb0f59be5
paper_authors.name: Xiuyu Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-12T03:05:21.524Z
paper_authors._id: 644570ba2d91b15b4c7f6311
paper_authors.avatarUrl: /avatars/d5e66012066d0c330b8f23718b1499d8.svg
paper_authors.fullname: Xiuyu Li
paper_authors.type: user
paper_authors.user: xiuyul
paper_authors._id: 661896d0de161bdcb0f59be6
paper_authors.name: Shishir Patil
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T12:30:43.149Z
paper_authors._id: 6471b9f6094820190c324eec
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6471b9f6094820190c324eec/yb62BiXzRFY1_pS0GAMGq.jpeg
paper_authors.fullname: Shishir Patil
paper_authors.type: user
paper_authors.user: shishirpatil
paper_authors._id: 661896d0de161bdcb0f59be7
paper_authors.name: Ziyang Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T12:31:39.967Z
paper_authors._id: 648fbaff4852c28c4cff7a3b
paper_authors.avatarUrl: /avatars/f987bb904580865676c8541839dd7615.svg
paper_authors.fullname: wuziyang
paper_authors.type: user
paper_authors.user: wzyohyes
paper_authors._id: 661896d0de161bdcb0f59be8
paper_authors.name: Tianjun Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T12:31:47.275Z
paper_authors._id: 6374cd6b6ea8da14f8fef8dc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6374cd6b6ea8da14f8fef8dc/l13bg0tKDjCnUw3I895QZ.png
paper_authors.fullname: Tianjun Zhang
paper_authors.type: user
paper_authors.user: tianjunz
paper_authors._id: 661896d0de161bdcb0f59be9
paper_authors.name: Kurt Keutzer
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T12:32:02.210Z
paper_authors._id: 6251bf4b183aa4266924ad91
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1678041834400-6251bf4b183aa4266924ad91.jpeg
paper_authors.fullname: Kurt Keutzer
paper_authors.type: user
paper_authors.user: kurtkeutzer
paper_authors._id: 661896d0de161bdcb0f59bea
paper_authors.name: Joseph E. Gonzalez
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T12:32:09.861Z
paper_authors._id: 645d2e8401f4eaab2a0878ce
paper_authors.avatarUrl: /avatars/1273c5fb607b4b622a746a42692fa632.svg
paper_authors.fullname: Joseph E. Gonzalez
paper_authors.type: user
paper_authors.user: ProfJoeyG
paper_authors._id: 661896d0de161bdcb0f59beb
paper_authors.name: Raluca Ada Popa
paper_title: LLoCO: Learning Long Contexts Offline
paper_summary: Processing long contexts remains a challenge for large language models (LLMs)
  due to the quadratic computational and memory overhead of the self-attention
  mechanism and the substantial KV cache sizes during generation. We propose a
  novel approach to address this problem by learning contexts offline through
  context compression and in-domain parameter-efficient finetuning. Our method
  enables an LLM to create a concise representation of the original context and
  efficiently retrieve relevant information to answer questions accurately. We
  introduce LLoCO, a technique that combines context compression, retrieval, and
  parameter-efficient finetuning using LoRA. Our approach extends the effective
  context window of a 4k token LLaMA2-7B model to handle up to 128k tokens. We
  evaluate our approach on several long-context question-answering datasets,
  demonstrating that LLoCO significantly outperforms in-context learning while
  using 30times fewer tokens during inference. LLoCO achieves up to
  7.62times speed-up and substantially reduces the cost of long document
  question answering, making it a promising solution for efficient long context
  processing. Our code is publicly available at
  https://github.com/jeffreysijuntan/lloco.
paper_ai_summary: LLoCO method enhances large language models' ability to handle long contexts efficiently by combining context compression, retrieval, and parameter-efficient finetuning.
paper_ai_keywords: self-attention context compression parameter-efficient finetuning LoRA LLaMA2-7B long-context question-answering in-context learning