LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic
  Prompt Compression
This paper focuses on task-agnostic prompt compression for better
generalizability and efficiency. Considering the redundancy in natural
language, existing approaches compress prompts by removing tokens or lexical
units according to their information entropy obtained from a causal language
model such as LLaMa-7B. The challenge is that information entropy may be a
suboptimal compression metric: (i) it only leverages unidirectional context and
may fail to capture all essential information needed for prompt compression;
(ii) it is not aligned with the prompt compression objective.
  To address these issues, we propose a data distillation procedure to derive
knowledge from an LLM to compress prompts without losing crucial information,
and meantime, introduce an extractive text compression dataset. We formulate
prompt compression as a token classification problem to guarantee the
faithfulness of the compressed prompt to the original one, and use a
Transformer encoder as the base architecture to capture all essential
information for prompt compression from the full bidirectional context. Our
approach leads to lower latency by explicitly learning the compression
objective with smaller models such as XLM-RoBERTa-large and mBERT.
  We evaluate our method on both in-domain and out-of-domain datasets,
including MeetingBank, LongBench, ZeroScrolls, GSM8K, and BBH. Despite its
small size, our model shows significant performance gains over strong baselines
and demonstrates robust generalization ability across different LLMs.
Additionally, our model is 3x-6x faster than existing prompt compression
methods, while accelerating the end-to-end latency by 1.6x-2.9x with
compression ratios of 2x-5x.
65fa5d220236fad825b90c14
Zhuoshi Pan
claimed_verified
2025-07-15T19:15:25.674Z
6565e44b9bf6665f10016213
/avatars/0d7cf0e1b42cd116960cd030478446c5.svg
Zhuoshi Pan
user
panzs19
65fa5d220236fad825b90c15
Qianhui Wu
admin_assigned
2024-03-21T12:03:19.190Z
63ef330b1e695b35aa484e11
https://cdn-avatars.huggingface.co/v1/production/uploads/63ef330b1e695b35aa484e11/bXwpGy0dl8JXeJwJ--ilr.jpeg
Qianhui WU
user
qianhuiwu
65fa5d220236fad825b90c16
Huiqiang Jiang
claimed_verified
2024-03-21T08:27:20.472Z
6278bd42541f3d2dfa77ea70
https://cdn-avatars.huggingface.co/v1/production/uploads/6278bd42541f3d2dfa77ea70/ejn49eapnB3UXQckAYdTd.jpeg
Huiqiang Jiang
user
iofu728
65fa5d220236fad825b90c17
Menglin Xia
65fa5d220236fad825b90c18
Xufang Luo
admin_assigned
2024-03-21T12:03:41.340Z
64b750a2fdb702b3d8619514
/avatars/f09181c0825763dff692c4bc65effc4c.svg
Xufang Luo
user
luoxufang
65fa5d220236fad825b90c19
Jue Zhang
admin_assigned
2024-03-21T12:03:47.912Z
65f40e43653c231cbaf7d1e4
/avatars/a42ac5454cbe175f04c3420fce90cad2.svg
Jue Zhang
user
JueZhang
65fa5d220236fad825b90c1a
Qingwei Lin
65fa5d220236fad825b90c1b
Victor Rühle
65fa5d220236fad825b90c1c
Yuqing Yang
65fa5d220236fad825b90c1d
Chin-Yew Lin
65fa5d220236fad825b90c1e
H. Vicky Zhao
65fa5d220236fad825b90c1f
Lili Qiu
65fa5d220236fad825b90c20
Dongmei Zhang
LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic
  Prompt Compression
This paper focuses on task-agnostic prompt compression for better
generalizability and efficiency. Considering the redundancy in natural
language, existing approaches compress prompts by removing tokens or lexical
units according to their information entropy obtained from a causal language
model such as LLaMa-7B. The challenge is that information entropy may be a
suboptimal compression metric: (i) it only leverages unidirectional context and
may fail to capture all essential information needed for prompt compression;
(ii) it is not aligned with the prompt compression objective.
  To address these issues, we propose a data distillation procedure to derive
knowledge from an LLM to compress prompts without losing crucial information,
and meantime, introduce an extractive text compression dataset. We formulate
prompt compression as a token classification problem to guarantee the
faithfulness of the compressed prompt to the original one, and use a
Transformer encoder as the base architecture to capture all essential
information for prompt compression from the full bidirectional context. Our
approach leads to lower latency by explicitly learning the compression
objective with smaller models such as XLM-RoBERTa-large and mBERT.
  We evaluate our method on both in-domain and out-of-domain datasets,
including MeetingBank, LongBench, ZeroScrolls, GSM8K, and BBH. Despite its
small size, our model shows significant performance gains over strong baselines
and demonstrates robust generalization ability across different LLMs.
Additionally, our model is 3x-6x faster than existing prompt compression
methods, while accelerating the end-to-end latency by 1.6x-2.9x with
compression ratios of 2x-5x.
A task-agnostic prompt compression method uses data distillation and a Transformer encoder to achieve efficient compression with minimal information loss and improved generalization.
prompt compression data distillation LLM token classification Transformer encoder XLM-RoBERTa mBERT MeetingBank LongBench ZeroScrolls GSM8K BBH