title: Reducing Transformer Key-Value Cache Size with Cross-Layer Attention
summary: Key-value (KV) caching plays an essential role in accelerating decoding for
  transformer-based autoregressive large language models (LLMs). However, the
  amount of memory required to store the KV cache can become prohibitive at long
  sequence lengths and large batch sizes. Since the invention of the transformer,
  two of the most effective interventions discovered for reducing the size of the
  KV cache have been Multi-Query Attention (MQA) and its generalization,
  Grouped-Query Attention (GQA). MQA and GQA both modify the design of the
  attention block so that multiple query heads can share a single key/value head,
  reducing the number of distinct key/value heads by a large factor while only
  minimally degrading accuracy. In this paper, we show that it is possible to
  take Multi-Query Attention a step further by also sharing key and value heads
  between adjacent layers, yielding a new attention design we call Cross-Layer
  Attention (CLA). With CLA, we find that it is possible to reduce the size of
  the KV cache by another 2x while maintaining nearly the same accuracy as
  unmodified MQA. In experiments training 1B- and 3B-parameter models from
  scratch, we demonstrate that CLA provides a Pareto improvement over the
  memory/accuracy tradeoffs which are possible with traditional MQA, enabling
  inference with longer sequence lengths and larger batch sizes than would
  otherwise be possible
paper_authors._id: 664d6cfe6ac7ea31242494eb
paper_authors.name: William Brandon
paper_authors._id: 664d6cfe6ac7ea31242494ec
paper_authors.name: Mayank Mishra
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-22T07:15:35.825Z
paper_authors._id: 62cd5057674cdb524450093d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62cd5057674cdb524450093d/f67rlrdsKPRTLdXCXoa_X.jpeg
paper_authors.fullname: Mayank Mishra
paper_authors.type: user
paper_authors.user: mayank-mishra
paper_authors._id: 664d6cfe6ac7ea31242494ed
paper_authors.name: Aniruddha Nrusimha
paper_authors._id: 664d6cfe6ac7ea31242494ee
paper_authors.name: Rameswar Panda
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-22T07:17:31.018Z
paper_authors._id: 648a7542ac1fbb8a1a654645
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/648a7542ac1fbb8a1a654645/jLPnY11EOuOa6ZHE7dVEX.jpeg
paper_authors.fullname: Rameswar Panda
paper_authors.type: user
paper_authors.user: rpand002
paper_authors._id: 664d6cfe6ac7ea31242494ef
paper_authors.name: Jonathan Ragan Kelly
paper_title: Reducing Transformer Key-Value Cache Size with Cross-Layer Attention
paper_summary: Key-value (KV) caching plays an essential role in accelerating decoding for
  transformer-based autoregressive large language models (LLMs). However, the
  amount of memory required to store the KV cache can become prohibitive at long
  sequence lengths and large batch sizes. Since the invention of the transformer,
  two of the most effective interventions discovered for reducing the size of the
  KV cache have been Multi-Query Attention (MQA) and its generalization,
  Grouped-Query Attention (GQA). MQA and GQA both modify the design of the
  attention block so that multiple query heads can share a single key/value head,
  reducing the number of distinct key/value heads by a large factor while only
  minimally degrading accuracy. In this paper, we show that it is possible to
  take Multi-Query Attention a step further by also sharing key and value heads
  between adjacent layers, yielding a new attention design we call Cross-Layer
  Attention (CLA). With CLA, we find that it is possible to reduce the size of
  the KV cache by another 2x while maintaining nearly the same accuracy as
  unmodified MQA. In experiments training 1B- and 3B-parameter models from
  scratch, we demonstrate that CLA provides a Pareto improvement over the
  memory/accuracy tradeoffs which are possible with traditional MQA, enabling
  inference with longer sequence lengths and larger batch sizes than would
  otherwise be possible
paper_ai_summary: Cross-Layer Attention modifies transformer-based autoregressive large language models to reduce KV cache size while maintaining accuracy, enabling longer sequence lengths and larger batch sizes during inference.
paper_ai_keywords: Key-value caching transformer-based autoregressive large language models MQA GQA Cross-Layer Attention