title: BitStack: Fine-Grained Size Control for Compressed Large Language Models
    in Variable Memory Environments
summary: Large language models (LLMs) have revolutionized numerous applications, yet
  their deployment remains challenged by memory constraints on local devices.
  While scaling laws have enhanced LLM capabilities, the primary bottleneck has
  shifted from capability to availability, emphasizing the need
  for efficient memory management. Traditional compression methods, such as
  quantization, often require predefined compression ratios and separate
  compression processes for each setting, complicating deployment in variable
  memory environments. In this paper, we introduce BitStack, a novel,
  training-free weight compression approach that enables megabyte-level
  trade-offs between memory usage and model performance. By leveraging weight
  decomposition, BitStack can dynamically adjust the model size with minimal
  transmission between running memory and storage devices. Our approach
  iteratively decomposes weight matrices while considering the significance of
  each parameter, resulting in an approximately 1-bit per parameter residual
  block in each decomposition iteration. These blocks are sorted and stacked in
  storage as basic transmission units, with different quantities loaded based on
  current memory availability. Extensive experiments across a wide range of tasks
  demonstrate that, despite offering fine-grained size control, BitStack
  consistently matches or surpasses strong quantization baselines, particularly
  at extreme compression ratios. To the best of our knowledge, this is the first
  decomposition-based method that effectively bridges the gap to practical
  compression techniques like quantization. Code is available at
  https://github.com/xinghaow99/BitStack.
paper_authors._id: 67248135b22109dd6c3c05de
paper_authors.name: Xinghao Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-03T14:21:54.150Z
paper_authors._id: 6191f22d08a57f265f7f5266
paper_authors.avatarUrl: /avatars/215164e7b6d025a3c32555ff541cdd62.svg
paper_authors.fullname: XinghaoWang
paper_authors.type: user
paper_authors.user: Singhoo
paper_authors._id: 67248135b22109dd6c3c05df
paper_authors.name: Pengyu Wang
paper_authors._id: 67248135b22109dd6c3c05e0
paper_authors.name: Bo Wang
paper_authors._id: 67248135b22109dd6c3c05e1
paper_authors.name: Dong Zhang
paper_authors._id: 67248135b22109dd6c3c05e2
paper_authors.name: Yunhua Zhou
paper_authors._id: 67248135b22109dd6c3c05e3
paper_authors.name: Xipeng Qiu
paper_title: BitStack: Fine-Grained Size Control for Compressed Large Language Models
    in Variable Memory Environments
paper_summary: Large language models (LLMs) have revolutionized numerous applications, yet
  their deployment remains challenged by memory constraints on local devices.
  While scaling laws have enhanced LLM capabilities, the primary bottleneck has
  shifted from capability to availability, emphasizing the need
  for efficient memory management. Traditional compression methods, such as
  quantization, often require predefined compression ratios and separate
  compression processes for each setting, complicating deployment in variable
  memory environments. In this paper, we introduce BitStack, a novel,
  training-free weight compression approach that enables megabyte-level
  trade-offs between memory usage and model performance. By leveraging weight
  decomposition, BitStack can dynamically adjust the model size with minimal
  transmission between running memory and storage devices. Our approach
  iteratively decomposes weight matrices while considering the significance of
  each parameter, resulting in an approximately 1-bit per parameter residual
  block in each decomposition iteration. These blocks are sorted and stacked in
  storage as basic transmission units, with different quantities loaded based on
  current memory availability. Extensive experiments across a wide range of tasks
  demonstrate that, despite offering fine-grained size control, BitStack
  consistently matches or surpasses strong quantization baselines, particularly
  at extreme compression ratios. To the best of our knowledge, this is the first
  decomposition-based method that effectively bridges the gap to practical
  compression techniques like quantization. Code is available at
  https://github.com/xinghaow99/BitStack.
paper_ai_summary: BitStack, a training-free weight compression approach, dynamically adjusts model size and memory usage with fine-grained control, outperforming traditional quantization methods.
paper_ai_keywords: BitStack weight compression weight decomposition residual block parameter significance fine-grained size control quantization memory usage model performance