title: Discovering the Gems in Early Layers: Accelerating Long-Context LLMs
    with 1000x Input Token Reduction
summary: Large Language Models (LLMs) have demonstrated remarkable capabilities in
  handling long context inputs, but this comes at the cost of increased
  computational resources and latency. Our research introduces a novel approach
  for the long context bottleneck to accelerate LLM inference and reduce GPU
  memory consumption. Our research demonstrates that LLMs can identify relevant
  tokens in the early layers before generating answers to a query. Leveraging
  this insight, we propose an algorithm that uses early layers of an LLM as
  filters to select and compress input tokens, significantly reducing the context
  length for subsequent processing. Our method, GemFilter, demonstrates
  substantial improvements in both speed and memory efficiency compared to
  existing techniques, such as standard attention and SnapKV/H2O. Notably, it
  achieves a 2.4times speedup and 30\% reduction in GPU memory usage compared
  to SOTA methods. Evaluation on the Needle in a Haystack task shows that
  GemFilter significantly outperforms standard attention, SnapKV and demonstrates
  comparable performance on the LongBench challenge. GemFilter is simple,
  training-free, and broadly applicable across different LLMs. Crucially, it
  provides interpretability by allowing humans to inspect the selected input
  sequence. These findings not only offer practical benefits for LLM deployment,
  but also enhance our understanding of LLM internal mechanisms, paving the way
  for further optimizations in LLM design and inference. Our code is available at
  https://github.com/SalesforceAIResearch/GemFilter.
paper_authors._id: 66f61c3e21cc430244ffbd46
paper_authors.name: Zhenmei Shi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-30T08:07:15.533Z
paper_authors._id: 64b769d7fa7eabaae5fb7f2f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64b769d7fa7eabaae5fb7f2f/AbxVphanApZDfj3cOI1tb.jpeg
paper_authors.fullname: Zhenmei Shi
paper_authors.type: user
paper_authors.user: Zhenmei
paper_authors._id: 66f61c3e21cc430244ffbd47
paper_authors.name: Yifei Ming
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-10-10T21:27:39.325Z
paper_authors._id: 641787ad1f1f3b0fa80f5340
paper_authors.avatarUrl: /avatars/489b7c999bce2160e27c9b487e1026ef.svg
paper_authors.fullname: yifei ming
paper_authors.type: user
paper_authors.user: alvinming
paper_authors._id: 66f61c3e21cc430244ffbd48
paper_authors.name: Xuan-Phi Nguyen
paper_authors._id: 66f61c3e21cc430244ffbd49
paper_authors.name: Yingyu Liang
paper_authors._id: 66f61c3e21cc430244ffbd4a
paper_authors.name: Shafiq Joty
paper_title: Discovering the Gems in Early Layers: Accelerating Long-Context LLMs
    with 1000x Input Token Reduction
paper_summary: Large Language Models (LLMs) have demonstrated remarkable capabilities in
  handling long context inputs, but this comes at the cost of increased
  computational resources and latency. Our research introduces a novel approach
  for the long context bottleneck to accelerate LLM inference and reduce GPU
  memory consumption. Our research demonstrates that LLMs can identify relevant
  tokens in the early layers before generating answers to a query. Leveraging
  this insight, we propose an algorithm that uses early layers of an LLM as
  filters to select and compress input tokens, significantly reducing the context
  length for subsequent processing. Our method, GemFilter, demonstrates
  substantial improvements in both speed and memory efficiency compared to
  existing techniques, such as standard attention and SnapKV/H2O. Notably, it
  achieves a 2.4times speedup and 30\% reduction in GPU memory usage compared
  to SOTA methods. Evaluation on the Needle in a Haystack task shows that
  GemFilter significantly outperforms standard attention, SnapKV and demonstrates
  comparable performance on the LongBench challenge. GemFilter is simple,
  training-free, and broadly applicable across different LLMs. Crucially, it
  provides interpretability by allowing humans to inspect the selected input
  sequence. These findings not only offer practical benefits for LLM deployment,
  but also enhance our understanding of LLM internal mechanisms, paving the way
  for further optimizations in LLM design and inference. Our code is available at
  https://github.com/SalesforceAIResearch/GemFilter.
paper_ai_keywords: Large Language Models (LLMs) long context inputs computational resources latency long context bottleneck LLM inference GPU memory consumption early layers input tokens filters context length standard attention SnapKV/H2O Needle in a Haystack task LongBench challenge training-free interpretability