title: LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inference
summary: The inference of transformer-based large language models consists of two
  sequential stages: 1) a prefilling stage to compute the KV cache of prompts and
  generate the first token, and 2) a decoding stage to generate subsequent
  tokens. For long prompts, the KV cache must be computed for all tokens during
  the prefilling stage, which can significantly increase the time needed to
  generate the first token. Consequently, the prefilling stage may become a
  bottleneck in the generation process. An open question remains whether all
  prompt tokens are essential for generating the first token. To answer this, we
  introduce a novel method, LazyLLM, that selectively computes the KV for tokens
  important for the next token prediction in both the prefilling and decoding
  stages. Contrary to static pruning approaches that prune the prompt at once,
  LazyLLM allows language models to dynamically select different subsets of
  tokens from the context in different generation steps, even though they might
  be pruned in previous steps. Extensive experiments on standard datasets across
  various tasks demonstrate that LazyLLM is a generic method that can be
  seamlessly integrated with existing language models to significantly accelerate
  the generation without fine-tuning. For instance, in the multi-document
  question-answering task, LazyLLM accelerates the prefilling stage of the LLama
  2 7B model by 2.34x while maintaining accuracy.
paper_authors._id: 669dc325dca4e06cb79e0c45
paper_authors.name: Qichen Fu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-22T07:51:24.541Z
paper_authors._id: 6496114d33611fb433f16970
paper_authors.avatarUrl: /avatars/ac722222c39e87c772705a3bf52118ad.svg
paper_authors.fullname: Qichen Fu
paper_authors.type: user
paper_authors.user: janemiminini
paper_authors._id: 669dc325dca4e06cb79e0c46
paper_authors.name: Minsik Cho
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-22T07:51:58.713Z
paper_authors._id: 630d422454c3dbd48058ba09
paper_authors.avatarUrl: /avatars/1985c5812523fddc0835cde18a490856.svg
paper_authors.fullname: minsik cho
paper_authors.type: user
paper_authors.user: thyeros
paper_authors._id: 669dc325dca4e06cb79e0c47
paper_authors.name: Thomas Merth
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-22T07:52:07.405Z
paper_authors._id: 64e67344a12333aea5fbf549
paper_authors.avatarUrl: /avatars/83b95306bf8232219a3fb9ee278740d7.svg
paper_authors.fullname: Thomas Merth
paper_authors.type: user
paper_authors.user: tmerthapple
paper_authors._id: 669dc325dca4e06cb79e0c48
paper_authors.name: Sachin Mehta
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-22T07:52:35.225Z
paper_authors._id: 64f792652003abb61bf5c1b8
paper_authors.avatarUrl: /avatars/9721810de3519ece50b059c154baa9ca.svg
paper_authors.fullname: Sachin Mehta
paper_authors.type: user
paper_authors.user: sacmehta
paper_authors._id: 669dc325dca4e06cb79e0c49
paper_authors.name: Mohammad Rastegari
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-22T07:52:41.061Z
paper_authors._id: 65d4c7734c2163d05eab4110
paper_authors.avatarUrl: /avatars/b116a9aca7b0a2a4836a29cb4f069d42.svg
paper_authors.fullname: Mohammad Rastegari
paper_authors.type: user
paper_authors.user: mrastegari
paper_authors._id: 669dc325dca4e06cb79e0c4a
paper_authors.name: Mahyar Najibi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-22T08:43:28.232Z
paper_authors._id: 661d992d69a65c0fb0717013
paper_authors.avatarUrl: /avatars/915d593a87111bca85e3500500246345.svg
paper_authors.fullname: Mahyar Najibi
paper_authors.type: user
paper_authors.user: mahyar-najibi
paper_title: LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inference
paper_summary: The inference of transformer-based large language models consists of two
  sequential stages: 1) a prefilling stage to compute the KV cache of prompts and
  generate the first token, and 2) a decoding stage to generate subsequent
  tokens. For long prompts, the KV cache must be computed for all tokens during
  the prefilling stage, which can significantly increase the time needed to
  generate the first token. Consequently, the prefilling stage may become a
  bottleneck in the generation process. An open question remains whether all
  prompt tokens are essential for generating the first token. To answer this, we
  introduce a novel method, LazyLLM, that selectively computes the KV for tokens
  important for the next token prediction in both the prefilling and decoding
  stages. Contrary to static pruning approaches that prune the prompt at once,
  LazyLLM allows language models to dynamically select different subsets of
  tokens from the context in different generation steps, even though they might
  be pruned in previous steps. Extensive experiments on standard datasets across
  various tasks demonstrate that LazyLLM is a generic method that can be
  seamlessly integrated with existing language models to significantly accelerate
  the generation without fine-tuning. For instance, in the multi-document
  question-answering task, LazyLLM accelerates the prefilling stage of the LLama
  2 7B model by 2.34x while maintaining accuracy.
paper_ai_keywords: transformer-based large language models KV cache prompts decoding stage prefilling stage LazyLLM dynamic selection context static pruning multi-document question-answering task LLama 2 7B model