title: InfiniteHiP: Extending Language Model Context Up to 3 Million Tokens on
    a Single GPU
summary: In modern large language models (LLMs), handling very long context lengths
  presents significant challenges as it causes slower inference speeds and
  increased memory costs. Additionally, most existing pre-trained LLMs fail to
  generalize beyond their original training sequence lengths. To enable efficient
  and practical long-context utilization, we introduce InfiniteHiP, a novel, and
  practical LLM inference framework that accelerates processing by dynamically
  eliminating irrelevant context tokens through a modular hierarchical token
  pruning algorithm. Our method also allows generalization to longer sequences by
  selectively applying various RoPE adjustment methods according to the internal
  attention patterns within LLMs. Furthermore, we offload the key-value cache to
  host memory during inference, significantly reducing GPU memory pressure. As a
  result, InfiniteHiP enables the processing of up to 3 million tokens on a
  single L40s 48GB GPU -- 3x larger -- without any permanent loss of context
  information. Our framework achieves an 18.95x speedup in attention decoding for
  a 1 million token context without requiring additional training. We implement
  our method in the SGLang framework and demonstrate its effectiveness and
  practicality through extensive evaluations.
paper_authors._id: 67aebd48225614bbe7f6f271
paper_authors.name: Heejun Lee
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-14T08:01:15.423Z
paper_authors._id: 62e622d08e0b2dc6707f8794
paper_authors.avatarUrl: /avatars/8c47b5c862f82d4258ba707c932f7f87.svg
paper_authors.fullname: Heejun Lee
paper_authors.type: user
paper_authors.user: gmlwns5176
paper_authors._id: 67aebd48225614bbe7f6f272
paper_authors.name: Geon Park
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-14T08:01:12.988Z
paper_authors._id: 646cae3093badbc8c2e891c7
paper_authors.avatarUrl: /avatars/4aae2aca70ea9dc58dd6f9f9b2be15e1.svg
paper_authors.fullname: Geon Park
paper_authors.type: user
paper_authors.user: geonp
paper_authors._id: 67aebd48225614bbe7f6f273
paper_authors.name: Jaduk Suh
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-14T12:00:46.888Z
paper_authors._id: 657ffd9ba2ef32167533f04a
paper_authors.avatarUrl: /avatars/e180e063c810c15d02b494727e962b84.svg
paper_authors.fullname: Jaduk Suh
paper_authors.type: user
paper_authors.user: Losif63
paper_authors._id: 67aebd48225614bbe7f6f274
paper_authors.name: Sung Ju Hwang
paper_title: InfiniteHiP: Extending Language Model Context Up to 3 Million Tokens on
    a Single GPU
paper_summary: In modern large language models (LLMs), handling very long context lengths
  presents significant challenges as it causes slower inference speeds and
  increased memory costs. Additionally, most existing pre-trained LLMs fail to
  generalize beyond their original training sequence lengths. To enable efficient
  and practical long-context utilization, we introduce InfiniteHiP, a novel, and
  practical LLM inference framework that accelerates processing by dynamically
  eliminating irrelevant context tokens through a modular hierarchical token
  pruning algorithm. Our method also allows generalization to longer sequences by
  selectively applying various RoPE adjustment methods according to the internal
  attention patterns within LLMs. Furthermore, we offload the key-value cache to
  host memory during inference, significantly reducing GPU memory pressure. As a
  result, InfiniteHiP enables the processing of up to 3 million tokens on a
  single L40s 48GB GPU -- 3x larger -- without any permanent loss of context
  information. Our framework achieves an 18.95x speedup in attention decoding for
  a 1 million token context without requiring additional training. We implement
  our method in the SGLang framework and demonstrate its effectiveness and
  practicality through extensive evaluations.
paper_ai_summary: InfiniteHiP is a novel LLM inference framework that enhances long-context processing with token pruning, RoPE adjustment, and host memory offloading, enabling faster and more memory-efficient operation.
paper_ai_keywords: LLMs InfiniteHiP token pruning hierarchical token pruning algorithm RoPE adjustment key-value cache host memory GPU memory pressure