title: SparQ Attention: Bandwidth-Efficient LLM Inference
summary: Generative large language models (LLMs) have opened up numerous novel
  possibilities, but due to their significant computational requirements their
  ubiquitous use remains challenging. Some of the most useful applications
  require processing large numbers of samples at a time and using long contexts,
  both significantly increasing the memory communication load of the models. We
  introduce SparQ Attention, a technique for increasing the inference throughput
  of LLMs by reducing the memory bandwidth requirements within the attention
  blocks through selective fetching of the cached history. Our proposed technique
  can be applied directly to off-the-shelf LLMs during inference, without
  requiring any modification to the pre-training setup or additional fine-tuning.
  We show how SparQ Attention can decrease the attention memory bandwidth
  requirements up to eight times without any loss in accuracy by evaluating Llama
  2 and Pythia models on a wide range of downstream tasks.
paper_authors._id: 65767c3fe09de6aa746884cf
paper_authors.name: Luka Ribar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-11T08:20:15.155Z
paper_authors._id: 65536094460d2d830c26ed6f
paper_authors.avatarUrl: /avatars/d1a088f69f96bf7b80778232904f9986.svg
paper_authors.fullname: Luka Ribar
paper_authors.type: user
paper_authors.user: lukaribar
paper_authors._id: 65767c3fe09de6aa746884d0
paper_authors.name: Ivan Chelombiev
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-11T08:20:21.785Z
paper_authors._id: 62c2b8b486741fbec3e69103
paper_authors.avatarUrl: /avatars/204953abced4edcdb80d29994611ba11.svg
paper_authors.fullname: Ivan Chelombiev
paper_authors.type: user
paper_authors.user: ivanc
paper_authors._id: 65767c3fe09de6aa746884d1
paper_authors.name: Luke Hudlass-Galley
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-11T08:20:28.082Z
paper_authors._id: 629095d8a29097b211b79cce
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/629095d8a29097b211b79cce/5bQnDw801V6C6YTwq_nGD.jpeg
paper_authors.fullname: Luke Hudlass-Galley
paper_authors.type: user
paper_authors.user: lukehg
paper_authors._id: 65767c3fe09de6aa746884d2
paper_authors.name: Charlie Blake
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-11T08:21:10.504Z
paper_authors._id: 64c978a1dc59e43a61146233
paper_authors.avatarUrl: /avatars/95a4a98d90079154316ca7a87ee22078.svg
paper_authors.fullname: Charlie Blake
paper_authors.type: user
paper_authors.user: ccbbb23
paper_authors._id: 65767c3fe09de6aa746884d3
paper_authors.name: Carlo Luschi
paper_authors._id: 65767c3fe09de6aa746884d4
paper_authors.name: Douglas Orr
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-11T08:21:52.063Z
paper_authors._id: 65364966f551a245bb049742
paper_authors.avatarUrl: /avatars/77caad0df6d5c1f667bb5b441f2aa0dc.svg
paper_authors.fullname: Douglas Orr
paper_authors.type: user
paper_authors.user: douglasorr
paper_title: SparQ Attention: Bandwidth-Efficient LLM Inference
paper_summary: Generative large language models (LLMs) have opened up numerous novel
  possibilities, but due to their significant computational requirements their
  ubiquitous use remains challenging. Some of the most useful applications
  require processing large numbers of samples at a time and using long contexts,
  both significantly increasing the memory communication load of the models. We
  introduce SparQ Attention, a technique for increasing the inference throughput
  of LLMs by reducing the memory bandwidth requirements within the attention
  blocks through selective fetching of the cached history. Our proposed technique
  can be applied directly to off-the-shelf LLMs during inference, without
  requiring any modification to the pre-training setup or additional fine-tuning.
  We show how SparQ Attention can decrease the attention memory bandwidth
  requirements up to eight times without any loss in accuracy by evaluating Llama
  2 and Pythia models on a wide range of downstream tasks.
paper_ai_keywords: SparQ Attention inference throughput memory bandwidth requirements attention blocks cached history Llama 2 Pythia models downstream tasks