title: Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic
    Capabilities in LLM Compression
summary: Post-training compression reduces the computational and memory costs of large
  language models (LLMs), enabling resource-efficient deployment. However,
  existing compression benchmarks only focus on language modeling (e.g.,
  perplexity) and natural language understanding tasks (e.g., GLUE accuracy),
  ignoring the agentic capabilities - workflow, tool use/function call,
  long-context understanding and real-world application. We introduce the Agent
  Compression Benchmark (ACBench), the first comprehensive benchmark for
  evaluating how compression impacts LLMs' agentic abilities. ACBench spans (1)
  12 tasks across 4 capabilities (e.g., WorfBench for workflow generation,
  Needle-in-Haystack for long-context retrieval), (2) quantization (GPTQ, AWQ)
  and pruning (Wanda, SparseGPT), and (3) 15 models, including small (Gemma-2B),
  standard (Qwen2.5 7B-32B), and distilled reasoning LLMs (DeepSeek-R1-Distill).
  Our experiments reveal compression tradeoffs: 4-bit quantization preserves
  workflow generation and tool use (1%-3% drop) but degrades real-world
  application accuracy by 10%-15%. We introduce ERank, Top-k Ranking Correlation
  and Energy to systematize analysis. ACBench provides actionable insights for
  optimizing LLM compression in agentic scenarios. The code can be found in
  https://github.com/pprp/ACBench.
paper_authors._id: 6836b4b68ec432fdc7e38251
paper_authors.name: Peijie Dong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-02T07:48:54.965Z
paper_authors._id: 6395f845aec00abff778ad31
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6395f845aec00abff778ad31/bZkAlchSvqER1HgBKmcHI.jpeg
paper_authors.fullname: PeijieDong
paper_authors.type: user
paper_authors.user: pprp
paper_authors._id: 6836b4b68ec432fdc7e38252
paper_authors.name: Zhenheng Tang
paper_authors._id: 6836b4b68ec432fdc7e38253
paper_authors.name: Xiang Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-28T12:12:19.286Z
paper_authors._id: 63024676056ec3a2a8714b24
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1661093436322-noauth.jpeg
paper_authors.fullname: Xiang Liu
paper_authors.type: user
paper_authors.user: Dominic789654
paper_authors._id: 6836b4b68ec432fdc7e38254
paper_authors.name: Lujun Li
paper_authors._id: 6836b4b68ec432fdc7e38255
paper_authors.name: Xiaowen Chu
paper_authors._id: 6836b4b68ec432fdc7e38256
paper_authors.name: Bo Li
paper_title: Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic
    Capabilities in LLM Compression
paper_summary: Post-training compression reduces the computational and memory costs of large
  language models (LLMs), enabling resource-efficient deployment. However,
  existing compression benchmarks only focus on language modeling (e.g.,
  perplexity) and natural language understanding tasks (e.g., GLUE accuracy),
  ignoring the agentic capabilities - workflow, tool use/function call,
  long-context understanding and real-world application. We introduce the Agent
  Compression Benchmark (ACBench), the first comprehensive benchmark for
  evaluating how compression impacts LLMs' agentic abilities. ACBench spans (1)
  12 tasks across 4 capabilities (e.g., WorfBench for workflow generation,
  Needle-in-Haystack for long-context retrieval), (2) quantization (GPTQ, AWQ)
  and pruning (Wanda, SparseGPT), and (3) 15 models, including small (Gemma-2B),
  standard (Qwen2.5 7B-32B), and distilled reasoning LLMs (DeepSeek-R1-Distill).
  Our experiments reveal compression tradeoffs: 4-bit quantization preserves
  workflow generation and tool use (1%-3% drop) but degrades real-world
  application accuracy by 10%-15%. We introduce ERank, Top-k Ranking Correlation
  and Energy to systematize analysis. ACBench provides actionable insights for
  optimizing LLM compression in agentic scenarios. The code can be found in
  https://github.com/pprp/ACBench.
paper_ai_summary: ACBench evaluates the impact of compression on the agentic capabilities of large language models, focusing on workflow generation, tool use, long-context understanding, and real-world application.
paper_ai_keywords: large language models LLMs post-training compression computational costs memory costs resource-efficient deployment language modeling perplexity natural language understanding GLUE accuracy Agent Compression Benchmark ACBench WorfBench Needle-in-Haystack quantization GPTQ AWQ pruning Wanda SparseGPT DeepSeek-R1-Distill ERank Top-k Ranking Correlation Energy agentic capabilities workflow generation tool use long-context understanding real-world application