title: MLKV: Multi-Layer Key-Value Heads for Memory Efficient Transformer
    Decoding
summary: Auto-regressive inference of transformers benefit greatly from Key-Value (KV)
  caching, but can lead to major memory bottlenecks as model size, batch size,
  and sequence length grow at scale. We introduce Multi-Layer Key-Value (MLKV)
  sharing, a novel approach extending KV sharing across transformer layers to
  reduce memory usage beyond what was possible with Multi-Query Attention (MQA)
  and Grouped-Query Attention (GQA). Evaluations on various NLP benchmarks and
  inference metrics using uptrained Pythia-160M variants demonstrate that MLKV
  significantly reduces memory usage with minimal performance loss, reducing KV
  cache size down to a factor of 6x compared to MQA. These results highlight
  MLKV's potential for efficient deployment of transformer models at scale. We
  provide code at https://github.com/zaydzuhri/pythia-mlkv
paper_authors._id: 666be7eef8649b8d8f562e90
paper_authors.name: Zayd Muhammad Kawakibi Zuhri
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-14T07:21:17.828Z
paper_authors._id: 60cf8a354061635e43b28f60
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60cf8a354061635e43b28f60/o8gv9mG5cvnopJZJnNibi.jpeg
paper_authors.fullname: Zayd Muhammad Kawakibi Zuhri
paper_authors.type: user
paper_authors.user: zaydzuhri
paper_authors._id: 666be7eef8649b8d8f562e91
paper_authors.name: Muhammad Farid Adilazuarda
paper_authors._id: 666be7eef8649b8d8f562e92
paper_authors.name: Ayu Purwarianti
paper_authors._id: 666be7eef8649b8d8f562e93
paper_authors.name: Alham Fikri Aji
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-14T13:56:01.718Z
paper_authors._id: 61a4dc053205e107691e0d82
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61a4dc053205e107691e0d82/BESoEHlHYXstXudh6dOdT.jpeg
paper_authors.fullname: Alham Fikri Aji
paper_authors.type: user
paper_authors.user: afaji
paper_title: MLKV: Multi-Layer Key-Value Heads for Memory Efficient Transformer
    Decoding
paper_summary: Auto-regressive inference of transformers benefit greatly from Key-Value (KV)
  caching, but can lead to major memory bottlenecks as model size, batch size,
  and sequence length grow at scale. We introduce Multi-Layer Key-Value (MLKV)
  sharing, a novel approach extending KV sharing across transformer layers to
  reduce memory usage beyond what was possible with Multi-Query Attention (MQA)
  and Grouped-Query Attention (GQA). Evaluations on various NLP benchmarks and
  inference metrics using uptrained Pythia-160M variants demonstrate that MLKV
  significantly reduces memory usage with minimal performance loss, reducing KV
  cache size down to a factor of 6x compared to MQA. These results highlight
  MLKV's potential for efficient deployment of transformer models at scale. We
  provide code at https://github.com/zaydzuhri/pythia-mlkv
paper_ai_summary: Multi-Layer Key-Value sharing in transformers reduces memory usage without significantly affecting performance, making large-scale deployment feasible.
paper_ai_keywords: auto-regressive inference transformers Key-Value caching Multi-Layer Key-Value sharing Multi-Query Attention Grouped-Query Attention KV cache size NLP benchmarks