SparQ Attention: Bandwidth-Efficient LLM Inference
Generative large language models (LLMs) have opened up numerous novel
possibilities, but due to their significant computational requirements their
ubiquitous use remains challenging. Some of the most useful applications
require processing large numbers of samples at a time and using long contexts,
both significantly increasing the memory communication load of the models. We
introduce SparQ Attention, a technique for increasing the inference throughput
of LLMs by reducing the memory bandwidth requirements within the attention
blocks through selective fetching of the cached history. Our proposed technique
can be applied directly to off-the-shelf LLMs during inference, without
requiring any modification to the pre-training setup or additional fine-tuning.
We show how SparQ Attention can decrease the attention memory bandwidth
requirements up to eight times without any loss in accuracy by evaluating Llama
2 and Pythia models on a wide range of downstream tasks.
65767c3fe09de6aa746884cf
Luka Ribar
admin_assigned
2023-12-11T08:20:15.155Z
65536094460d2d830c26ed6f
/avatars/d1a088f69f96bf7b80778232904f9986.svg
Luka Ribar
user
lukaribar
65767c3fe09de6aa746884d0
Ivan Chelombiev
admin_assigned
2023-12-11T08:20:21.785Z
62c2b8b486741fbec3e69103
/avatars/204953abced4edcdb80d29994611ba11.svg
Ivan Chelombiev
user
ivanc
65767c3fe09de6aa746884d1
Luke Hudlass-Galley
admin_assigned
2023-12-11T08:20:28.082Z
629095d8a29097b211b79cce
https://cdn-avatars.huggingface.co/v1/production/uploads/629095d8a29097b211b79cce/5bQnDw801V6C6YTwq_nGD.jpeg
Luke Hudlass-Galley
user
lukehg
65767c3fe09de6aa746884d2
Charlie Blake
admin_assigned
2023-12-11T08:21:10.504Z
64c978a1dc59e43a61146233
/avatars/95a4a98d90079154316ca7a87ee22078.svg
Charlie Blake
user
ccbbb23
65767c3fe09de6aa746884d3
Carlo Luschi
65767c3fe09de6aa746884d4
Douglas Orr
admin_assigned
2023-12-11T08:21:52.063Z
65364966f551a245bb049742
/avatars/77caad0df6d5c1f667bb5b441f2aa0dc.svg
Douglas Orr
user
douglasorr
SparQ Attention: Bandwidth-Efficient LLM Inference
Generative large language models (LLMs) have opened up numerous novel
possibilities, but due to their significant computational requirements their
ubiquitous use remains challenging. Some of the most useful applications
require processing large numbers of samples at a time and using long contexts,
both significantly increasing the memory communication load of the models. We
introduce SparQ Attention, a technique for increasing the inference throughput
of LLMs by reducing the memory bandwidth requirements within the attention
blocks through selective fetching of the cached history. Our proposed technique
can be applied directly to off-the-shelf LLMs during inference, without
requiring any modification to the pre-training setup or additional fine-tuning.
We show how SparQ Attention can decrease the attention memory bandwidth
requirements up to eight times without any loss in accuracy by evaluating Llama
2 and Pythia models on a wide range of downstream tasks.
SparQ Attention reduces memory bandwidth requirements in LLM attention blocks, enhancing inference throughput without accuracy loss.
SparQ Attention generative large language models (LLMs) inference throughput attention blocks memory bandwidth Llama 2 Pythia models downstream tasks