title: 70% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU
    Inference via Dynamic-Length Float
summary: Large Language Models (LLMs) have grown rapidly in size, creating significant
  challenges for efficient deployment on resource-constrained hardware. In this
  paper, we introduce Dynamic-Length Float (DFloat11), a lossless compression
  framework that reduces LLM size by 30% while preserving outputs that are
  bit-for-bit identical to the original model. DFloat11 is motivated by the low
  entropy in the BFloat16 weight representation of LLMs, which reveals
  significant inefficiency in existing storage format. By applying entropy
  coding, DFloat11 assigns dynamic-length encodings to weights based on
  frequency, achieving near information-optimal compression without any loss of
  precision. To facilitate efficient inference with dynamic-length encodings, we
  develop a custom GPU kernel for fast online decompression. Our design
  incorporates the following: (i) decomposition of memory-intensive lookup tables
  (LUTs) into compact LUTs that fit in GPU SRAM, (ii) a two-phase kernel for
  coordinating thread read/write positions using lightweight auxiliary variables,
  and (iii) transformer-block-level decompression to minimize latency.
  Experiments on recent models, including Llama-3.1, Qwen-2.5, and Gemma-3,
  validates our hypothesis that DFloat11 achieves around 30% model size reduction
  while preserving bit-for-bit exact outputs. Compared to a potential alternative
  of offloading parts of an uncompressed model to the CPU to meet memory
  constraints, DFloat11 achieves 1.9-38.8x higher throughput in token generation.
  With a fixed GPU memory budget, DFloat11 enables 5.3-13.17x longer context
  lengths than uncompressed models. Notably, our method enables lossless
  inference of Llama-3.1-405B, an 810GB model, on a single node equipped with
  8x80GB GPUs. Our code and models are available at
  https://github.com/LeanModels/DFloat11.
paper_authors._id: 6801228e8b6f40490fb179f8
paper_authors.name: Tianyi Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-18T09:39:45.219Z
paper_authors._id: 66a3196b98016fc5e1731bfb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/66a3196b98016fc5e1731bfb/Kx3IdfxIXOl4U-ox7CoxT.png
paper_authors.fullname: Tianyi Zhang
paper_authors.type: user
paper_authors.user: LeanQuant
paper_authors._id: 6801228e8b6f40490fb179f9
paper_authors.name: Yang Sui
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-20T15:03:33.167Z
paper_authors._id: 63787b13500186f250ba377c
paper_authors.avatarUrl: /avatars/96bb6051662109e9cd25e6df7d738f17.svg
paper_authors.fullname: yangsui
paper_authors.type: user
paper_authors.user: yangsui
paper_authors._id: 6801228e8b6f40490fb179fa
paper_authors.name: Shaochen Zhong
paper_authors._id: 6801228e8b6f40490fb179fb
paper_authors.name: Vipin Chaudhary
paper_authors._id: 6801228e8b6f40490fb179fc
paper_authors.name: Xia Hu
paper_authors._id: 6801228e8b6f40490fb179fd
paper_authors.name: Anshumali Shrivastava
paper_title: 70% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU
    Inference via Dynamic-Length Float
paper_summary: Large Language Models (LLMs) have grown rapidly in size, creating significant
  challenges for efficient deployment on resource-constrained hardware. In this
  paper, we introduce Dynamic-Length Float (DFloat11), a lossless compression
  framework that reduces LLM size by 30% while preserving outputs that are
  bit-for-bit identical to the original model. DFloat11 is motivated by the low
  entropy in the BFloat16 weight representation of LLMs, which reveals
  significant inefficiency in existing storage format. By applying entropy
  coding, DFloat11 assigns dynamic-length encodings to weights based on
  frequency, achieving near information-optimal compression without any loss of
  precision. To facilitate efficient inference with dynamic-length encodings, we
  develop a custom GPU kernel for fast online decompression. Our design
  incorporates the following: (i) decomposition of memory-intensive lookup tables
  (LUTs) into compact LUTs that fit in GPU SRAM, (ii) a two-phase kernel for
  coordinating thread read/write positions using lightweight auxiliary variables,
  and (iii) transformer-block-level decompression to minimize latency.
  Experiments on recent models, including Llama-3.1, Qwen-2.5, and Gemma-3,
  validates our hypothesis that DFloat11 achieves around 30% model size reduction
  while preserving bit-for-bit exact outputs. Compared to a potential alternative
  of offloading parts of an uncompressed model to the CPU to meet memory
  constraints, DFloat11 achieves 1.9-38.8x higher throughput in token generation.
  With a fixed GPU memory budget, DFloat11 enables 5.3-13.17x longer context
  lengths than uncompressed models. Notably, our method enables lossless
  inference of Llama-3.1-405B, an 810GB model, on a single node equipped with
  8x80GB GPUs. Our code and models are available at
  https://github.com/LeanModels/DFloat11.
paper_ai_summary: DFloat11, a lossless compression framework, reduces Large Language Model sizes by 30% through entropy coding, enabling efficient deployment and significantly higher throughput on resource-constrained hardware.
paper_ai_keywords: Dynamic-Length Float DFloat11 LLM Large Language Models lossless compression entropy coding BFloat16 weights GPU kernel memory-intensive lookup tables LUTs GPU SRAM transformer-block-level decompression token generation context lengths