InfiniPot: Infinite Context Processing on Memory-Constrained LLMs
Handling long input contexts remains a significant challenge for Large
Language Models (LLMs), particularly in resource-constrained environments such
as mobile devices. Our work aims to address this limitation by introducing
InfiniPot, a novel KV cache control framework designed to enable pre-trained
LLMs to manage extensive sequences within fixed memory constraints efficiently,
without requiring additional training. InfiniPot leverages Continual Context
Distillation (CCD), an iterative process that compresses and retains essential
information through novel importance metrics, effectively maintaining critical
data even without access to future context. Our comprehensive evaluations
indicate that InfiniPot significantly outperforms models trained for long
contexts in various NLP tasks, establishing its efficacy and versatility. This
work represents a substantial advancement toward making LLMs applicable to a
broader range of real-world scenarios.
66fea1fbbb54785c35647ede
Minsoo Kim
claimed_verified
2024-10-03T15:01:41.434Z
63c0e2503bdc86f8108da51b
/avatars/7d47f11992f030b3d831e45102581d1f.svg
Minsoo Kim
user
minsoo2333
66fea1fbbb54785c35647edf
Kyuhong Shim
66fea1fbbb54785c35647ee0
Jungwook Choi
66fea1fbbb54785c35647ee1
Simyung Chang
InfiniPot: Infinite Context Processing on Memory-Constrained LLMs
Handling long input contexts remains a significant challenge for Large
Language Models (LLMs), particularly in resource-constrained environments such
as mobile devices. Our work aims to address this limitation by introducing
InfiniPot, a novel KV cache control framework designed to enable pre-trained
LLMs to manage extensive sequences within fixed memory constraints efficiently,
without requiring additional training. InfiniPot leverages Continual Context
Distillation (CCD), an iterative process that compresses and retains essential
information through novel importance metrics, effectively maintaining critical
data even without access to future context. Our comprehensive evaluations
indicate that InfiniPot significantly outperforms models trained for long
contexts in various NLP tasks, establishing its efficacy and versatility. This
work represents a substantial advancement toward making LLMs applicable to a
broader range of real-world scenarios.
InfiniPot, a new KV cache control framework using Continual Context Distillation, allows LLMs to handle long input contexts efficiently within limited memory.
KV cache Large Language Models (LLMs) Continual Context Distillation (CCD) importance metrics