title: InfiniPot: Infinite Context Processing on Memory-Constrained LLMs
summary: Handling long input contexts remains a significant challenge for Large
  Language Models (LLMs), particularly in resource-constrained environments such
  as mobile devices. Our work aims to address this limitation by introducing
  InfiniPot, a novel KV cache control framework designed to enable pre-trained
  LLMs to manage extensive sequences within fixed memory constraints efficiently,
  without requiring additional training. InfiniPot leverages Continual Context
  Distillation (CCD), an iterative process that compresses and retains essential
  information through novel importance metrics, effectively maintaining critical
  data even without access to future context. Our comprehensive evaluations
  indicate that InfiniPot significantly outperforms models trained for long
  contexts in various NLP tasks, establishing its efficacy and versatility. This
  work represents a substantial advancement toward making LLMs applicable to a
  broader range of real-world scenarios.
paper_authors._id: 66fea1fbbb54785c35647ede
paper_authors.name: Minsoo Kim
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-03T15:01:41.434Z
paper_authors._id: 63c0e2503bdc86f8108da51b
paper_authors.avatarUrl: /avatars/7d47f11992f030b3d831e45102581d1f.svg
paper_authors.fullname: Minsoo Kim
paper_authors.type: user
paper_authors.user: minsoo2333
paper_authors._id: 66fea1fbbb54785c35647edf
paper_authors.name: Kyuhong Shim
paper_authors._id: 66fea1fbbb54785c35647ee0
paper_authors.name: Jungwook Choi
paper_authors._id: 66fea1fbbb54785c35647ee1
paper_authors.name: Simyung Chang
paper_title: InfiniPot: Infinite Context Processing on Memory-Constrained LLMs
paper_summary: Handling long input contexts remains a significant challenge for Large
  Language Models (LLMs), particularly in resource-constrained environments such
  as mobile devices. Our work aims to address this limitation by introducing
  InfiniPot, a novel KV cache control framework designed to enable pre-trained
  LLMs to manage extensive sequences within fixed memory constraints efficiently,
  without requiring additional training. InfiniPot leverages Continual Context
  Distillation (CCD), an iterative process that compresses and retains essential
  information through novel importance metrics, effectively maintaining critical
  data even without access to future context. Our comprehensive evaluations
  indicate that InfiniPot significantly outperforms models trained for long
  contexts in various NLP tasks, establishing its efficacy and versatility. This
  work represents a substantial advancement toward making LLMs applicable to a
  broader range of real-world scenarios.
paper_ai_keywords: KV cache control framework Continual Context Distillation (CCD) importance metrics