title: Characterizing Prompt Compression Methods for Long Context Inference
summary: Long context inference presents challenges at the system level with increased
  compute and memory requirements, as well as from an accuracy perspective in
  being able to reason over long contexts. Recently, several methods have been
  proposed to compress the prompt to reduce the context length. However, there
  has been little work on comparing the different proposed methods across
  different tasks through a standardized analysis. This has led to conflicting
  results. To address this, here we perform a comprehensive characterization and
  evaluation of different prompt compression methods. In particular, we analyze
  extractive compression, summarization-based abstractive compression, and token
  pruning methods. Surprisingly, we find that extractive compression often
  outperforms all the other approaches, and enables up to 10x compression with
  minimal accuracy degradation. Interestingly, we also find that despite several
  recent claims, token pruning methods often lag behind extractive compression.
  We only found marginal improvements on summarization tasks.
paper_authors._id: 66948ea3eb7d71adc7547180
paper_authors.name: Siddharth Jha
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-15T08:58:24.805Z
paper_authors._id: 646fc09c850a938d6c54c2eb
paper_authors.avatarUrl: /avatars/f4793a7367d372def34014551db48be2.svg
paper_authors.fullname: Sid Jha
paper_authors.type: user
paper_authors.user: sidjha1
paper_authors._id: 66948ea3eb7d71adc7547181
paper_authors.name: Lutfi Eren Erdogan
paper_authors._id: 66948ea3eb7d71adc7547182
paper_authors.name: Sehoon Kim
paper_authors._id: 66948ea3eb7d71adc7547183
paper_authors.name: Kurt Keutzer
paper_authors._id: 66948ea3eb7d71adc7547184
paper_authors.name: Amir Gholami
paper_title: Characterizing Prompt Compression Methods for Long Context Inference
paper_summary: Long context inference presents challenges at the system level with increased
  compute and memory requirements, as well as from an accuracy perspective in
  being able to reason over long contexts. Recently, several methods have been
  proposed to compress the prompt to reduce the context length. However, there
  has been little work on comparing the different proposed methods across
  different tasks through a standardized analysis. This has led to conflicting
  results. To address this, here we perform a comprehensive characterization and
  evaluation of different prompt compression methods. In particular, we analyze
  extractive compression, summarization-based abstractive compression, and token
  pruning methods. Surprisingly, we find that extractive compression often
  outperforms all the other approaches, and enables up to 10x compression with
  minimal accuracy degradation. Interestingly, we also find that despite several
  recent claims, token pruning methods often lag behind extractive compression.
  We only found marginal improvements on summarization tasks.
paper_ai_summary: The study evaluates various prompt compression methods, finding extractive compression to outperform others, enabling significant context reduction with minimal accuracy loss.
paper_ai_keywords: extractive compression summarization-based abstractive compression token pruning prompt compression long context inference context length