Characterizing Prompt Compression Methods for Long Context Inference
Long context inference presents challenges at the system level with increased
compute and memory requirements, as well as from an accuracy perspective in
being able to reason over long contexts. Recently, several methods have been
proposed to compress the prompt to reduce the context length. However, there
has been little work on comparing the different proposed methods across
different tasks through a standardized analysis. This has led to conflicting
results. To address this, here we perform a comprehensive characterization and
evaluation of different prompt compression methods. In particular, we analyze
extractive compression, summarization-based abstractive compression, and token
pruning methods. Surprisingly, we find that extractive compression often
outperforms all the other approaches, and enables up to 10x compression with
minimal accuracy degradation. Interestingly, we also find that despite several
recent claims, token pruning methods often lag behind extractive compression.
We only found marginal improvements on summarization tasks.
66948ea3eb7d71adc7547180
Siddharth Jha
claimed_verified
2024-07-15T08:58:24.805Z
646fc09c850a938d6c54c2eb
/avatars/f4793a7367d372def34014551db48be2.svg
Sid Jha
user
sidjha1
66948ea3eb7d71adc7547181
Lutfi Eren Erdogan
66948ea3eb7d71adc7547182
Sehoon Kim
66948ea3eb7d71adc7547183
Kurt Keutzer
66948ea3eb7d71adc7547184
Amir Gholami
Characterizing Prompt Compression Methods for Long Context Inference
Long context inference presents challenges at the system level with increased
compute and memory requirements, as well as from an accuracy perspective in
being able to reason over long contexts. Recently, several methods have been
proposed to compress the prompt to reduce the context length. However, there
has been little work on comparing the different proposed methods across
different tasks through a standardized analysis. This has led to conflicting
results. To address this, here we perform a comprehensive characterization and
evaluation of different prompt compression methods. In particular, we analyze
extractive compression, summarization-based abstractive compression, and token
pruning methods. Surprisingly, we find that extractive compression often
outperforms all the other approaches, and enables up to 10x compression with
minimal accuracy degradation. Interestingly, we also find that despite several
recent claims, token pruning methods often lag behind extractive compression.
We only found marginal improvements on summarization tasks.
The study evaluates various prompt compression methods, finding extractive compression to outperform others, enabling significant context reduction with minimal accuracy loss.
extractive compression summarization-based abstractive compression token pruning prompt compression long context inference context length