Discovering the Gems in Early Layers: Accelerating Long-Context LLMs
  with 1000x Input Token Reduction
Large Language Models (LLMs) have demonstrated remarkable capabilities in
handling long context inputs, but this comes at the cost of increased
computational resources and latency. Our research introduces a novel approach
for the long context bottleneck to accelerate LLM inference and reduce GPU
memory consumption. Our research demonstrates that LLMs can identify relevant
tokens in the early layers before generating answers to a query. Leveraging
this insight, we propose an algorithm that uses early layers of an LLM as
filters to select and compress input tokens, significantly reducing the context
length for subsequent processing. Our method, GemFilter, demonstrates
substantial improvements in both speed and memory efficiency compared to
existing techniques, such as standard attention and SnapKV/H2O. Notably, it
achieves a 2.4times speedup and 30\% reduction in GPU memory usage compared
to SOTA methods. Evaluation on the Needle in a Haystack task shows that
GemFilter significantly outperforms standard attention, SnapKV and demonstrates
comparable performance on the LongBench challenge. GemFilter is simple,
training-free, and broadly applicable across different LLMs. Crucially, it
provides interpretability by allowing humans to inspect the selected input
sequence. These findings not only offer practical benefits for LLM deployment,
but also enhance our understanding of LLM internal mechanisms, paving the way
for further optimizations in LLM design and inference. Our code is available at
https://github.com/SalesforceAIResearch/GemFilter.
66f61c3e21cc430244ffbd46
Zhenmei Shi
claimed_verified
2024-09-30T08:07:15.533Z
64b769d7fa7eabaae5fb7f2f
https://cdn-avatars.huggingface.co/v1/production/uploads/64b769d7fa7eabaae5fb7f2f/AbxVphanApZDfj3cOI1tb.jpeg
Zhenmei Shi
user
Zhenmei
66f61c3e21cc430244ffbd47
Yifei Ming
extracted_confirmed
2024-10-10T21:27:39.325Z
641787ad1f1f3b0fa80f5340
/avatars/489b7c999bce2160e27c9b487e1026ef.svg
yifei ming
user
alvinming
66f61c3e21cc430244ffbd48
Xuan-Phi Nguyen
66f61c3e21cc430244ffbd49
Yingyu Liang
66f61c3e21cc430244ffbd4a
Shafiq Joty
Discovering the Gems in Early Layers: Accelerating Long-Context LLMs
  with 1000x Input Token Reduction
Large Language Models (LLMs) have demonstrated remarkable capabilities in
handling long context inputs, but this comes at the cost of increased
computational resources and latency. Our research introduces a novel approach
for the long context bottleneck to accelerate LLM inference and reduce GPU
memory consumption. Our research demonstrates that LLMs can identify relevant
tokens in the early layers before generating answers to a query. Leveraging
this insight, we propose an algorithm that uses early layers of an LLM as
filters to select and compress input tokens, significantly reducing the context
length for subsequent processing. Our method, GemFilter, demonstrates
substantial improvements in both speed and memory efficiency compared to
existing techniques, such as standard attention and SnapKV/H2O. Notably, it
achieves a 2.4times speedup and 30\% reduction in GPU memory usage compared
to SOTA methods. Evaluation on the Needle in a Haystack task shows that
GemFilter significantly outperforms standard attention, SnapKV and demonstrates
comparable performance on the LongBench challenge. GemFilter is simple,
training-free, and broadly applicable across different LLMs. Crucially, it
provides interpretability by allowing humans to inspect the selected input
sequence. These findings not only offer practical benefits for LLM deployment,
but also enhance our understanding of LLM internal mechanisms, paving the way
for further optimizations in LLM design and inference. Our code is available at
https://github.com/SalesforceAIResearch/GemFilter.
GemFilter algorithm accelerates Large Language Model inference and reduces GPU memory usage by filtering input tokens using early model layers.
Large Language Models (LLMs) long context bottleneck LLM inference GemFilter token selection early layers speedup GPU memory usage Needle in a Haystack task LongBench challenge interpretability