title: LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic
    Prompt Compression
summary: This paper focuses on task-agnostic prompt compression for better
  generalizability and efficiency. Considering the redundancy in natural
  language, existing approaches compress prompts by removing tokens or lexical
  units according to their information entropy obtained from a causal language
  model such as LLaMa-7B. The challenge is that information entropy may be a
  suboptimal compression metric: (i) it only leverages unidirectional context and
  may fail to capture all essential information needed for prompt compression;
  (ii) it is not aligned with the prompt compression objective.
    To address these issues, we propose a data distillation procedure to derive
  knowledge from an LLM to compress prompts without losing crucial information,
  and meantime, introduce an extractive text compression dataset. We formulate
  prompt compression as a token classification problem to guarantee the
  faithfulness of the compressed prompt to the original one, and use a
  Transformer encoder as the base architecture to capture all essential
  information for prompt compression from the full bidirectional context. Our
  approach leads to lower latency by explicitly learning the compression
  objective with smaller models such as XLM-RoBERTa-large and mBERT.
    We evaluate our method on both in-domain and out-of-domain datasets,
  including MeetingBank, LongBench, ZeroScrolls, GSM8K, and BBH. Despite its
  small size, our model shows significant performance gains over strong baselines
  and demonstrates robust generalization ability across different LLMs.
  Additionally, our model is 3x-6x faster than existing prompt compression
  methods, while accelerating the end-to-end latency by 1.6x-2.9x with
  compression ratios of 2x-5x.
paper_authors._id: 65fa5d220236fad825b90c14
paper_authors.name: Zhuoshi Pan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-21T12:03:12.680Z
paper_authors._id: 6565e24fe5aac326bfd15a9d
paper_authors.avatarUrl: /avatars/28ad90df0e0dbc10ef25ee6499a50dec.svg
paper_authors.fullname: Zhuoshi Pan
paper_authors.type: user
paper_authors.user: panzs
paper_authors._id: 65fa5d220236fad825b90c15
paper_authors.name: Qianhui Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-21T12:03:19.190Z
paper_authors._id: 63ef330b1e695b35aa484e11
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63ef330b1e695b35aa484e11/bXwpGy0dl8JXeJwJ--ilr.jpeg
paper_authors.fullname: Qianhui WU
paper_authors.type: user
paper_authors.user: qianhuiwu
paper_authors._id: 65fa5d220236fad825b90c16
paper_authors.name: Huiqiang Jiang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-21T08:27:20.472Z
paper_authors._id: 6278bd42541f3d2dfa77ea70
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6278bd42541f3d2dfa77ea70/ejn49eapnB3UXQckAYdTd.jpeg
paper_authors.fullname: Huiqiang Jiang
paper_authors.type: user
paper_authors.user: iofu728
paper_authors._id: 65fa5d220236fad825b90c17
paper_authors.name: Menglin Xia
paper_authors._id: 65fa5d220236fad825b90c18
paper_authors.name: Xufang Luo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-21T12:03:41.340Z
paper_authors._id: 64b750a2fdb702b3d8619514
paper_authors.avatarUrl: /avatars/f09181c0825763dff692c4bc65effc4c.svg
paper_authors.fullname: Xufang Luo
paper_authors.type: user
paper_authors.user: luoxufang
paper_authors._id: 65fa5d220236fad825b90c19
paper_authors.name: Jue Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-21T12:03:47.912Z
paper_authors._id: 65f40e43653c231cbaf7d1e4
paper_authors.avatarUrl: /avatars/a42ac5454cbe175f04c3420fce90cad2.svg
paper_authors.fullname: Jue Zhang
paper_authors.type: user
paper_authors.user: JueZhang
paper_authors._id: 65fa5d220236fad825b90c1a
paper_authors.name: Qingwei Lin
paper_authors._id: 65fa5d220236fad825b90c1b
paper_authors.name: Victor Rühle
paper_authors._id: 65fa5d220236fad825b90c1c
paper_authors.name: Yuqing Yang
paper_authors._id: 65fa5d220236fad825b90c1d
paper_authors.name: Chin-Yew Lin
paper_authors._id: 65fa5d220236fad825b90c1e
paper_authors.name: H. Vicky Zhao
paper_authors._id: 65fa5d220236fad825b90c1f
paper_authors.name: Lili Qiu
paper_authors._id: 65fa5d220236fad825b90c20
paper_authors.name: Dongmei Zhang
paper_title: LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic
    Prompt Compression
paper_summary: This paper focuses on task-agnostic prompt compression for better
  generalizability and efficiency. Considering the redundancy in natural
  language, existing approaches compress prompts by removing tokens or lexical
  units according to their information entropy obtained from a causal language
  model such as LLaMa-7B. The challenge is that information entropy may be a
  suboptimal compression metric: (i) it only leverages unidirectional context and
  may fail to capture all essential information needed for prompt compression;
  (ii) it is not aligned with the prompt compression objective.
    To address these issues, we propose a data distillation procedure to derive
  knowledge from an LLM to compress prompts without losing crucial information,
  and meantime, introduce an extractive text compression dataset. We formulate
  prompt compression as a token classification problem to guarantee the
  faithfulness of the compressed prompt to the original one, and use a
  Transformer encoder as the base architecture to capture all essential
  information for prompt compression from the full bidirectional context. Our
  approach leads to lower latency by explicitly learning the compression
  objective with smaller models such as XLM-RoBERTa-large and mBERT.
    We evaluate our method on both in-domain and out-of-domain datasets,
  including MeetingBank, LongBench, ZeroScrolls, GSM8K, and BBH. Despite its
  small size, our model shows significant performance gains over strong baselines
  and demonstrates robust generalization ability across different LLMs.
  Additionally, our model is 3x-6x faster than existing prompt compression
  methods, while accelerating the end-to-end latency by 1.6x-2.9x with
  compression ratios of 2x-5x.
paper_ai_keywords: causal language model information entropy prompt compression data distillation extractive text compression token classification Transformer encoder XLM-RoBERTa-large mBERT MeetingBank LongBench ZeroScrolls GSM8K BBH end-to-end latency compression ratios