BitStack: Fine-Grained Size Control for Compressed Large Language Models
  in Variable Memory Environments
Large language models (LLMs) have revolutionized numerous applications, yet
their deployment remains challenged by memory constraints on local devices.
While scaling laws have enhanced LLM capabilities, the primary bottleneck has
shifted from capability to availability, emphasizing the need
for efficient memory management. Traditional compression methods, such as
quantization, often require predefined compression ratios and separate
compression processes for each setting, complicating deployment in variable
memory environments. In this paper, we introduce BitStack, a novel,
training-free weight compression approach that enables megabyte-level
trade-offs between memory usage and model performance. By leveraging weight
decomposition, BitStack can dynamically adjust the model size with minimal
transmission between running memory and storage devices. Our approach
iteratively decomposes weight matrices while considering the significance of
each parameter, resulting in an approximately 1-bit per parameter residual
block in each decomposition iteration. These blocks are sorted and stacked in
storage as basic transmission units, with different quantities loaded based on
current memory availability. Extensive experiments across a wide range of tasks
demonstrate that, despite offering fine-grained size control, BitStack
consistently matches or surpasses strong quantization baselines, particularly
at extreme compression ratios. To the best of our knowledge, this is the first
decomposition-based method that effectively bridges the gap to practical
compression techniques like quantization. Code is available at
https://github.com/xinghaow99/BitStack.
67248135b22109dd6c3c05de
Xinghao Wang
claimed_verified
2024-11-03T14:21:54.150Z
6191f22d08a57f265f7f5266
/avatars/215164e7b6d025a3c32555ff541cdd62.svg
XinghaoWang
user
Singhoo
67248135b22109dd6c3c05df
Pengyu Wang
67248135b22109dd6c3c05e0
Bo Wang
67248135b22109dd6c3c05e1
Dong Zhang
67248135b22109dd6c3c05e2
Yunhua Zhou
67248135b22109dd6c3c05e3
Xipeng Qiu
BitStack: Fine-Grained Size Control for Compressed Large Language Models
  in Variable Memory Environments
Large language models (LLMs) have revolutionized numerous applications, yet
their deployment remains challenged by memory constraints on local devices.
While scaling laws have enhanced LLM capabilities, the primary bottleneck has
shifted from capability to availability, emphasizing the need
for efficient memory management. Traditional compression methods, such as
quantization, often require predefined compression ratios and separate
compression processes for each setting, complicating deployment in variable
memory environments. In this paper, we introduce BitStack, a novel,
training-free weight compression approach that enables megabyte-level
trade-offs between memory usage and model performance. By leveraging weight
decomposition, BitStack can dynamically adjust the model size with minimal
transmission between running memory and storage devices. Our approach
iteratively decomposes weight matrices while considering the significance of
each parameter, resulting in an approximately 1-bit per parameter residual
block in each decomposition iteration. These blocks are sorted and stacked in
storage as basic transmission units, with different quantities loaded based on
current memory availability. Extensive experiments across a wide range of tasks
demonstrate that, despite offering fine-grained size control, BitStack
consistently matches or surpasses strong quantization baselines, particularly
at extreme compression ratios. To the best of our knowledge, this is the first
decomposition-based method that effectively bridges the gap to practical
compression techniques like quantization. Code is available at
https://github.com/xinghaow99/BitStack.
BitStack, a training-free weight compression approach, dynamically adjusts model size and memory usage with fine-grained control, outperforming traditional quantization methods.
BitStack weight compression weight decomposition residual block parameter significance fine-grained size control quantization memory usage model performance