Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time
Large language models (LLMs) with hundreds of billions of parameters have
sparked a new wave of exciting AI applications. However, they are
computationally expensive at inference time. Sparsity is a natural approach to
reduce this cost, but existing methods either require costly retraining, have
to forgo LLM's in-context learning ability, or do not yield wall-clock time
speedup on modern hardware. We hypothesize that contextual sparsity, which are
small, input-dependent sets of attention heads and MLP parameters that yield
approximately the same output as the dense model for a given input, can address
these issues. We show that contextual sparsity exists, that it can be
accurately predicted, and that we can exploit it to speed up LLM inference in
wall-clock time without compromising LLM's quality or in-context learning
ability. Based on these insights, we propose DejaVu, a system that uses a
low-cost algorithm to predict contextual sparsity on the fly given inputs to
each layer, along with an asynchronous and hardware-aware implementation that
speeds up LLM inference. We validate that DejaVu can reduce the inference
latency of OPT-175B by over 2X compared to the state-of-the-art
FasterTransformer, and over 6X compared to the widely used Hugging Face
implementation, without compromising model quality. The code is available at
https://github.com/FMInference/DejaVu.
653b22cc42bfd8801c4a4d06
Zichang Liu
admin_assigned
2023-10-27T09:21:11.579Z
65081166338e4879e47360f6
/avatars/3564c1ea0d87c2433ca6f3a6f43b6256.svg
Zichang Liu
user
freeliuzc
653b22cc42bfd8801c4a4d07
Jue Wang
admin_assigned
2023-10-27T09:21:56.488Z
61dce5c2af6d5e733e0fb08b
https://cdn-avatars.huggingface.co/v1/production/uploads/1669380535028-61dce5c2af6d5e733e0fb08b.jpeg
Jue Wang
user
juewang
653b22cc42bfd8801c4a4d08
Tri Dao
admin_assigned
2023-10-27T09:22:04.472Z
64b8a6b5cf14c2fabe98159b
/avatars/dbc009451865435bf290791beadc4723.svg
Tri Dao
user
tridao
653b22cc42bfd8801c4a4d09
Tianyi Zhou
653b22cc42bfd8801c4a4d0a
Binhang Yuan
admin_assigned
2023-10-27T09:23:39.555Z
6307c8dda642348dc5e71c20
https://cdn-avatars.huggingface.co/v1/production/uploads/1661454675064-6307c8dda642348dc5e71c20.jpeg
Binhang Yuan
user
biyuan
653b22cc42bfd8801c4a4d0b
Zhao Song
653b22cc42bfd8801c4a4d0c
Anshumali Shrivastava
admin_assigned
2023-10-27T09:24:08.379Z
641b4d4fabfce26bcf7b6f4a
/avatars/482588424a590e7e04f7d830de206174.svg
Anshumali Shrivastava
user
anshumali
653b22cc42bfd8801c4a4d0d
Ce Zhang
653b22cc42bfd8801c4a4d0e
Yuandong Tian
admin_assigned
2023-10-27T09:24:28.399Z
6344cf73ee1504dbcd5bdfe7
/avatars/6dd2bf1f9c5679e5c8c85d62c9836aac.svg
Yuandong Tian
user
tydsh
653b22cc42bfd8801c4a4d0f
Christopher Re
653b22cc42bfd8801c4a4d10
Beidi Chen
admin_assigned
2023-10-27T09:24:45.046Z
64b732f832403871593e082c
/avatars/dd21932b0c167131ee7545a622c46c3c.svg
Beidi Chen
user
beidic
Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time
Large language models (LLMs) with hundreds of billions of parameters have
sparked a new wave of exciting AI applications. However, they are
computationally expensive at inference time. Sparsity is a natural approach to
reduce this cost, but existing methods either require costly retraining, have
to forgo LLM's in-context learning ability, or do not yield wall-clock time
speedup on modern hardware. We hypothesize that contextual sparsity, which are
small, input-dependent sets of attention heads and MLP parameters that yield
approximately the same output as the dense model for a given input, can address
these issues. We show that contextual sparsity exists, that it can be
accurately predicted, and that we can exploit it to speed up LLM inference in
wall-clock time without compromising LLM's quality or in-context learning
ability. Based on these insights, we propose DejaVu, a system that uses a
low-cost algorithm to predict contextual sparsity on the fly given inputs to
each layer, along with an asynchronous and hardware-aware implementation that
speeds up LLM inference. We validate that DejaVu can reduce the inference
latency of OPT-175B by over 2X compared to the state-of-the-art
FasterTransformer, and over 6X compared to the widely used Hugging Face
implementation, without compromising model quality. The code is available at
https://github.com/FMInference/DejaVu.
DejaVu system predicts and exploits contextual sparsity on large language models to significantly reduce inference latency without affecting quality.
large language models LLM inference time sparsity attention heads MLP parameters contextual sparsity FasterTransformer Hugging Face implementation inference latency model quality