title: Decoding Compressed Trust: Scrutinizing the Trustworthiness of Efficient
    LLMs Under Compression
summary: Compressing high-capability Large Language Models (LLMs) has emerged as a
  favored strategy for resource-efficient inferences. While state-of-the-art
  (SoTA) compression methods boast impressive advancements in preserving benign
  task performance, the potential risks of compression in terms of safety and
  trustworthiness have been largely neglected. This study conducts the first,
  thorough evaluation of three (3) leading LLMs using five (5) SoTA compression
  techniques across eight (8) trustworthiness dimensions. Our experiments
  highlight the intricate interplay between compression and trustworthiness,
  revealing some interesting patterns. We find that quantization is currently a
  more effective approach than pruning in achieving efficiency and
  trustworthiness simultaneously. For instance, a 4-bit quantized model retains
  the trustworthiness of its original counterpart, but model pruning
  significantly degrades trustworthiness, even at 50% sparsity. Moreover,
  employing quantization within a moderate bit range could unexpectedly improve
  certain trustworthiness dimensions such as ethics and fairness. Conversely,
  extreme quantization to very low bit levels (3 bits) tends to significantly
  reduce trustworthiness. This increased risk cannot be uncovered by looking at
  benign performance alone, in turn, mandating comprehensive trustworthiness
  evaluation in practice. These findings culminate in practical recommendations
  for simultaneously achieving high utility, efficiency, and trustworthiness in
  LLMs. Models and code are available at https://decoding-comp-trust.github.io/.
paper_authors._id: 66024c5f73c7ce480391b863
paper_authors.name: Junyuan Hong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-26T07:54:17.032Z
paper_authors._id: 6400cf982b67d27affce2d89
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6400cf982b67d27affce2d89/Vs042d2M-iV2wk9Q_Jqh3.jpeg
paper_authors.fullname: Junyuan Hong
paper_authors.type: user
paper_authors.user: jyhong836
paper_authors._id: 66024c5f73c7ce480391b864
paper_authors.name: Jinhao Duan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-26T13:58:51.787Z
paper_authors._id: 6423855c2d5965e7229ebe57
paper_authors.avatarUrl: /avatars/babb01f5f2329cc95e13e33b1ec5cf25.svg
paper_authors.fullname: jinhaoduan
paper_authors.type: user
paper_authors.user: jhao
paper_authors._id: 66024c5f73c7ce480391b865
paper_authors.name: Chenhui Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-26T07:54:19.015Z
paper_authors._id: 64522233ea94bf023430dd95
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/CVDqDeJ_fLTULhCTTSogb.png
paper_authors.fullname: Chenhui Zhang
paper_authors.type: user
paper_authors.user: danielz01
paper_authors._id: 66024c5f73c7ce480391b866
paper_authors.name: Zhangheng Li
paper_authors._id: 66024c5f73c7ce480391b867
paper_authors.name: Chulin Xie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-26T09:12:24.428Z
paper_authors._id: 645867e3ead43697df2ca5fb
paper_authors.avatarUrl: /avatars/a0f0e5d975898413be456667c0cbd2a8.svg
paper_authors.fullname: Chulin Xie
paper_authors.type: user
paper_authors.user: alphapav
paper_authors._id: 66024c5f73c7ce480391b868
paper_authors.name: Kelsey Lieberman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-26T09:12:30.611Z
paper_authors._id: 633eeebf0ffccea68bd7c4c7
paper_authors.avatarUrl: /avatars/4959ad22f0f4eaf5712ba7a38f74679a.svg
paper_authors.fullname: Kelsey Lieberman
paper_authors.type: user
paper_authors.user: klieberman
paper_authors._id: 66024c5f73c7ce480391b869
paper_authors.name: James Diffenderfer
paper_authors._id: 66024c5f73c7ce480391b86a
paper_authors.name: Brian Bartoldson
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-26T09:13:19.748Z
paper_authors._id: 641a7ecb91e3376a057adf89
paper_authors.avatarUrl: /avatars/f407adae3e9df964a469cfb350115eb8.svg
paper_authors.fullname: brian bartoldson
paper_authors.type: user
paper_authors.user: bartoldson
paper_authors._id: 66024c5f73c7ce480391b86b
paper_authors.name: Ajay Jaiswal
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-26T09:13:26.229Z
paper_authors._id: 64ad7afdafb6aa5534221c96
paper_authors.avatarUrl: /avatars/c1c83d8f5ca683ce60fd4d50c028471c.svg
paper_authors.fullname: Ajay Jaiswal
paper_authors.type: user
paper_authors.user: Ajay1994
paper_authors._id: 66024c5f73c7ce480391b86c
paper_authors.name: Kaidi Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-26T09:14:11.603Z
paper_authors._id: 65aaa5b4d2adc31ee3eab350
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65aaa5b4d2adc31ee3eab350/vzDrR7JVCEShn_Vk-NCgT.jpeg
paper_authors.fullname: Kaidi Xu
paper_authors.type: user
paper_authors.user: KaidiXu1
paper_authors._id: 66024c5f73c7ce480391b86d
paper_authors.name: Bhavya Kailkhura
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-26T09:19:33.378Z
paper_authors._id: 65cb79db6427380bc21261e2
paper_authors.avatarUrl: /avatars/a003eb5d0955417329c1a4170ae65879.svg
paper_authors.fullname: Bhavya Kailkhura
paper_authors.type: user
paper_authors.user: bhavyakailkhura
paper_authors._id: 66024c5f73c7ce480391b86e
paper_authors.name: Dan Hendrycks
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-26T09:19:39.622Z
paper_authors._id: 63f55aa1b51da4d61da9c96b
paper_authors.avatarUrl: /avatars/56cf9c2d8295c4549248d3b0a4933043.svg
paper_authors.fullname: Dan Hendrycks
paper_authors.type: user
paper_authors.user: hendrycks
paper_authors._id: 66024c5f73c7ce480391b86f
paper_authors.name: Dawn Song
paper_authors._id: 66024c5f73c7ce480391b870
paper_authors.name: Zhangyang Wang
paper_authors._id: 66024c5f73c7ce480391b871
paper_authors.name: Bo Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-26T09:20:48.218Z
paper_authors._id: 6493236b70d925ae8050a1bf
paper_authors.avatarUrl: /avatars/b16069de1445cfa8608567175deaa2ae.svg
paper_authors.fullname: Bo Li
paper_authors.type: user
paper_authors.user: BoLi-aisecure
paper_title: Decoding Compressed Trust: Scrutinizing the Trustworthiness of Efficient
    LLMs Under Compression
paper_summary: Compressing high-capability Large Language Models (LLMs) has emerged as a
  favored strategy for resource-efficient inferences. While state-of-the-art
  (SoTA) compression methods boast impressive advancements in preserving benign
  task performance, the potential risks of compression in terms of safety and
  trustworthiness have been largely neglected. This study conducts the first,
  thorough evaluation of three (3) leading LLMs using five (5) SoTA compression
  techniques across eight (8) trustworthiness dimensions. Our experiments
  highlight the intricate interplay between compression and trustworthiness,
  revealing some interesting patterns. We find that quantization is currently a
  more effective approach than pruning in achieving efficiency and
  trustworthiness simultaneously. For instance, a 4-bit quantized model retains
  the trustworthiness of its original counterpart, but model pruning
  significantly degrades trustworthiness, even at 50% sparsity. Moreover,
  employing quantization within a moderate bit range could unexpectedly improve
  certain trustworthiness dimensions such as ethics and fairness. Conversely,
  extreme quantization to very low bit levels (3 bits) tends to significantly
  reduce trustworthiness. This increased risk cannot be uncovered by looking at
  benign performance alone, in turn, mandating comprehensive trustworthiness
  evaluation in practice. These findings culminate in practical recommendations
  for simultaneously achieving high utility, efficiency, and trustworthiness in
  LLMs. Models and code are available at https://decoding-comp-trust.github.io/.
paper_ai_summary: Compression techniques, particularly quantization, have varying impacts on the trustworthiness of large language models, with quantization within moderate bit ranges enhancing certain ethical dimensions while extreme quantization degrades trustworthiness.
paper_ai_keywords: Large Language Models LLMs compression techniques quantization pruning trustworthiness ethics fairness