title: Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time
summary: Large language models (LLMs) with hundreds of billions of parameters have
  sparked a new wave of exciting AI applications. However, they are
  computationally expensive at inference time. Sparsity is a natural approach to
  reduce this cost, but existing methods either require costly retraining, have
  to forgo LLM's in-context learning ability, or do not yield wall-clock time
  speedup on modern hardware. We hypothesize that contextual sparsity, which are
  small, input-dependent sets of attention heads and MLP parameters that yield
  approximately the same output as the dense model for a given input, can address
  these issues. We show that contextual sparsity exists, that it can be
  accurately predicted, and that we can exploit it to speed up LLM inference in
  wall-clock time without compromising LLM's quality or in-context learning
  ability. Based on these insights, we propose DejaVu, a system that uses a
  low-cost algorithm to predict contextual sparsity on the fly given inputs to
  each layer, along with an asynchronous and hardware-aware implementation that
  speeds up LLM inference. We validate that DejaVu can reduce the inference
  latency of OPT-175B by over 2X compared to the state-of-the-art
  FasterTransformer, and over 6X compared to the widely used Hugging Face
  implementation, without compromising model quality. The code is available at
  https://github.com/FMInference/DejaVu.
paper_authors._id: 653b22cc42bfd8801c4a4d06
paper_authors.name: Zichang Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-27T09:21:11.579Z
paper_authors._id: 65081166338e4879e47360f6
paper_authors.avatarUrl: /avatars/3564c1ea0d87c2433ca6f3a6f43b6256.svg
paper_authors.fullname: Zichang Liu
paper_authors.type: user
paper_authors.user: freeliuzc
paper_authors._id: 653b22cc42bfd8801c4a4d07
paper_authors.name: Jue Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-27T09:21:56.488Z
paper_authors._id: 61dce5c2af6d5e733e0fb08b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1669380535028-61dce5c2af6d5e733e0fb08b.jpeg
paper_authors.fullname: Jue Wang
paper_authors.type: user
paper_authors.user: juewang
paper_authors._id: 653b22cc42bfd8801c4a4d08
paper_authors.name: Tri Dao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-27T09:22:04.472Z
paper_authors._id: 64b8a6b5cf14c2fabe98159b
paper_authors.avatarUrl: /avatars/dbc009451865435bf290791beadc4723.svg
paper_authors.fullname: Tri Dao
paper_authors.type: user
paper_authors.user: tridao
paper_authors._id: 653b22cc42bfd8801c4a4d09
paper_authors.name: Tianyi Zhou
paper_authors._id: 653b22cc42bfd8801c4a4d0a
paper_authors.name: Binhang Yuan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-27T09:23:39.555Z
paper_authors._id: 6307c8dda642348dc5e71c20
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1661454675064-6307c8dda642348dc5e71c20.jpeg
paper_authors.fullname: Binhang Yuan
paper_authors.type: user
paper_authors.user: biyuan
paper_authors._id: 653b22cc42bfd8801c4a4d0b
paper_authors.name: Zhao Song
paper_authors._id: 653b22cc42bfd8801c4a4d0c
paper_authors.name: Anshumali Shrivastava
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-27T09:24:08.379Z
paper_authors._id: 641b4d4fabfce26bcf7b6f4a
paper_authors.avatarUrl: /avatars/482588424a590e7e04f7d830de206174.svg
paper_authors.fullname: Anshumali Shrivastava
paper_authors.type: user
paper_authors.user: anshumali
paper_authors._id: 653b22cc42bfd8801c4a4d0d
paper_authors.name: Ce Zhang
paper_authors._id: 653b22cc42bfd8801c4a4d0e
paper_authors.name: Yuandong Tian
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-27T09:24:28.399Z
paper_authors._id: 6344cf73ee1504dbcd5bdfe7
paper_authors.avatarUrl: /avatars/6dd2bf1f9c5679e5c8c85d62c9836aac.svg
paper_authors.fullname: Yuandong Tian
paper_authors.type: user
paper_authors.user: tydsh
paper_authors._id: 653b22cc42bfd8801c4a4d0f
paper_authors.name: Christopher Re
paper_authors._id: 653b22cc42bfd8801c4a4d10
paper_authors.name: Beidi Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-27T09:24:45.046Z
paper_authors._id: 64b732f832403871593e082c
paper_authors.avatarUrl: /avatars/dd21932b0c167131ee7545a622c46c3c.svg
paper_authors.fullname: Beidi Chen
paper_authors.type: user
paper_authors.user: beidic
paper_title: Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time
paper_summary: Large language models (LLMs) with hundreds of billions of parameters have
  sparked a new wave of exciting AI applications. However, they are
  computationally expensive at inference time. Sparsity is a natural approach to
  reduce this cost, but existing methods either require costly retraining, have
  to forgo LLM's in-context learning ability, or do not yield wall-clock time
  speedup on modern hardware. We hypothesize that contextual sparsity, which are
  small, input-dependent sets of attention heads and MLP parameters that yield
  approximately the same output as the dense model for a given input, can address
  these issues. We show that contextual sparsity exists, that it can be
  accurately predicted, and that we can exploit it to speed up LLM inference in
  wall-clock time without compromising LLM's quality or in-context learning
  ability. Based on these insights, we propose DejaVu, a system that uses a
  low-cost algorithm to predict contextual sparsity on the fly given inputs to
  each layer, along with an asynchronous and hardware-aware implementation that
  speeds up LLM inference. We validate that DejaVu can reduce the inference
  latency of OPT-175B by over 2X compared to the state-of-the-art
  FasterTransformer, and over 6X compared to the widely used Hugging Face
  implementation, without compromising model quality. The code is available at
  https://github.com/FMInference/DejaVu.
paper_ai_keywords: Large Language Models (LLMs) in-context learning attention heads MLP parameters contextual sparsity DejaVu asynchronous implementation hardware-aware implementation