title: SwiftKV: Fast Prefill-Optimized Inference with Knowledge-Preserving
    Model Transformation
summary: LLM inference for popular enterprise use cases, such as summarization, RAG,
  and code-generation, typically observes orders of magnitude longer prompt
  lengths than generation lengths. This characteristic leads to high cost of
  prefill and increased response latency. In this paper, we present SwiftKV, a
  novel model transformation and distillation procedure specifically designed to
  reduce the time and cost of processing prompt tokens while preserving high
  quality of generated tokens. SwiftKV combines three key mechanisms: i)
  SingleInputKV, which prefills later layers' KV cache using a much earlier
  layer's output, allowing prompt tokens to skip much of the model computation,
  ii) AcrossKV, which merges the KV caches of neighboring layers to reduce the
  memory footprint and support larger batch size for higher throughput, and iii)
  a knowledge-preserving distillation procedure that can adapt existing LLMs for
  SwiftKV with minimal accuracy impact and low compute and data requirement. For
  Llama-3.1-8B and 70B, SwiftKV reduces the compute requirement of prefill by 50%
  and the memory requirement of the KV cache by 62.5% while incurring minimum
  quality degradation across a wide range of tasks. In the end-to-end inference
  serving using an optimized vLLM implementation, SwiftKV realizes up to 2x
  higher aggregate throughput and 60% lower time per output token. It can achieve
  a staggering 560 TFlops/GPU of normalized inference throughput, which
  translates to 16K tokens/s for Llama-3.1-70B in 16-bit precision on 4x H100
  GPUs.
paper_authors._id: 67056753ec0355056ef05fc9
paper_authors.name: Aurick Qiao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-09T13:14:58.787Z
paper_authors._id: 630e8f0bf6f6d700f50ebd2e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/630e8f0bf6f6d700f50ebd2e/ELJAinK1XpVJ8aslZ3BIW.jpeg
paper_authors.fullname: Aurick Qiao
paper_authors.type: user
paper_authors.user: aurick
paper_authors._id: 67056753ec0355056ef05fca
paper_authors.name: Zhewei Yao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-09T13:15:05.347Z
paper_authors._id: 6352e1953199c2612ec2ff01
paper_authors.avatarUrl: /avatars/f75444f1b7cba63ed085774a950a3fc7.svg
paper_authors.fullname: Zhewei Yao
paper_authors.type: user
paper_authors.user: zheweiyao
paper_authors._id: 67056753ec0355056ef05fcb
paper_authors.name: Samyam Rajbhandari
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-09T13:14:44.165Z
paper_authors._id: 63238e79d444e1d90888ea03
paper_authors.avatarUrl: /avatars/cd0accb5ddec4f2ae23c94da4094b5e5.svg
paper_authors.fullname: Samyam Rajbhandari
paper_authors.type: user
paper_authors.user: samyam
paper_authors._id: 67056753ec0355056ef05fcc
paper_authors.name: Yuxiong He
paper_title: SwiftKV: Fast Prefill-Optimized Inference with Knowledge-Preserving
    Model Transformation
paper_summary: LLM inference for popular enterprise use cases, such as summarization, RAG,
  and code-generation, typically observes orders of magnitude longer prompt
  lengths than generation lengths. This characteristic leads to high cost of
  prefill and increased response latency. In this paper, we present SwiftKV, a
  novel model transformation and distillation procedure specifically designed to
  reduce the time and cost of processing prompt tokens while preserving high
  quality of generated tokens. SwiftKV combines three key mechanisms: i)
  SingleInputKV, which prefills later layers' KV cache using a much earlier
  layer's output, allowing prompt tokens to skip much of the model computation,
  ii) AcrossKV, which merges the KV caches of neighboring layers to reduce the
  memory footprint and support larger batch size for higher throughput, and iii)
  a knowledge-preserving distillation procedure that can adapt existing LLMs for
  SwiftKV with minimal accuracy impact and low compute and data requirement. For
  Llama-3.1-8B and 70B, SwiftKV reduces the compute requirement of prefill by 50%
  and the memory requirement of the KV cache by 62.5% while incurring minimum
  quality degradation across a wide range of tasks. In the end-to-end inference
  serving using an optimized vLLM implementation, SwiftKV realizes up to 2x
  higher aggregate throughput and 60% lower time per output token. It can achieve
  a staggering 560 TFlops/GPU of normalized inference throughput, which
  translates to 16K tokens/s for Llama-3.1-70B in 16-bit precision on 4x H100
  GPUs.
paper_ai_summary: SwiftKV optimizes LLM inference by reducing prefill compute and memory usage without degrading quality, leading to higher throughput and lower latency in enterprise use cases.
paper_ai_keywords: SingleInputKV AcrossKV knowledge-preserving distillation vLLM TFlops inference throughput