LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inference
The inference of transformer-based large language models consists of two
sequential stages: 1) a prefilling stage to compute the KV cache of prompts and
generate the first token, and 2) a decoding stage to generate subsequent
tokens. For long prompts, the KV cache must be computed for all tokens during
the prefilling stage, which can significantly increase the time needed to
generate the first token. Consequently, the prefilling stage may become a
bottleneck in the generation process. An open question remains whether all
prompt tokens are essential for generating the first token. To answer this, we
introduce a novel method, LazyLLM, that selectively computes the KV for tokens
important for the next token prediction in both the prefilling and decoding
stages. Contrary to static pruning approaches that prune the prompt at once,
LazyLLM allows language models to dynamically select different subsets of
tokens from the context in different generation steps, even though they might
be pruned in previous steps. Extensive experiments on standard datasets across
various tasks demonstrate that LazyLLM is a generic method that can be
seamlessly integrated with existing language models to significantly accelerate
the generation without fine-tuning. For instance, in the multi-document
question-answering task, LazyLLM accelerates the prefilling stage of the LLama
2 7B model by 2.34x while maintaining accuracy.
669dc325dca4e06cb79e0c45
Qichen Fu
admin_assigned
2024-07-22T07:51:24.541Z
6496114d33611fb433f16970
/avatars/ac722222c39e87c772705a3bf52118ad.svg
Qichen Fu
user
janemiminini
669dc325dca4e06cb79e0c46
Minsik Cho
admin_assigned
2024-07-22T07:51:58.713Z
630d422454c3dbd48058ba09
/avatars/1985c5812523fddc0835cde18a490856.svg
minsik cho
user
thyeros
669dc325dca4e06cb79e0c47
Thomas Merth
admin_assigned
2024-07-22T07:52:07.405Z
64e67344a12333aea5fbf549
/avatars/83b95306bf8232219a3fb9ee278740d7.svg
Thomas Merth
user
tmerthapple
669dc325dca4e06cb79e0c48
Sachin Mehta
admin_assigned
2024-07-22T07:52:35.225Z
64f792652003abb61bf5c1b8
/avatars/9721810de3519ece50b059c154baa9ca.svg
Sachin Mehta
user
sacmehta
669dc325dca4e06cb79e0c49
Mohammad Rastegari
admin_assigned
2024-07-22T07:52:41.061Z
65d4c7734c2163d05eab4110
/avatars/b116a9aca7b0a2a4836a29cb4f069d42.svg
Mohammad Rastegari
user
mrastegari
669dc325dca4e06cb79e0c4a
Mahyar Najibi
admin_assigned
2024-07-22T08:43:28.232Z
661d992d69a65c0fb0717013
/avatars/9d1f20baff742de90a50232db54d2407.svg
Mahyar Najibi
user
mahyar-najibi
LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inference
The inference of transformer-based large language models consists of two
sequential stages: 1) a prefilling stage to compute the KV cache of prompts and
generate the first token, and 2) a decoding stage to generate subsequent
tokens. For long prompts, the KV cache must be computed for all tokens during
the prefilling stage, which can significantly increase the time needed to
generate the first token. Consequently, the prefilling stage may become a
bottleneck in the generation process. An open question remains whether all
prompt tokens are essential for generating the first token. To answer this, we
introduce a novel method, LazyLLM, that selectively computes the KV for tokens
important for the next token prediction in both the prefilling and decoding
stages. Contrary to static pruning approaches that prune the prompt at once,
LazyLLM allows language models to dynamically select different subsets of
tokens from the context in different generation steps, even though they might
be pruned in previous steps. Extensive experiments on standard datasets across
various tasks demonstrate that LazyLLM is a generic method that can be
seamlessly integrated with existing language models to significantly accelerate
the generation without fine-tuning. For instance, in the multi-document
question-answering task, LazyLLM accelerates the prefilling stage of the LLama
2 7B model by 2.34x while maintaining accuracy.
LazyLLM dynamically selects important prompt tokens to accelerate the prefilling stage in transformer-based language models without sacrificing accuracy.
transformer-based large language models KV cache prefilling stage decoding stage LazyLLM static pruning approaches multi-document question-answering LLama 2 7B model