70% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU
  Inference via Dynamic-Length Float
Large Language Models (LLMs) have grown rapidly in size, creating significant
challenges for efficient deployment on resource-constrained hardware. In this
paper, we introduce Dynamic-Length Float (DFloat11), a lossless compression
framework that reduces LLM size by 30% while preserving outputs that are
bit-for-bit identical to the original model. DFloat11 is motivated by the low
entropy in the BFloat16 weight representation of LLMs, which reveals
significant inefficiency in existing storage format. By applying entropy
coding, DFloat11 assigns dynamic-length encodings to weights based on
frequency, achieving near information-optimal compression without any loss of
precision. To facilitate efficient inference with dynamic-length encodings, we
develop a custom GPU kernel for fast online decompression. Our design
incorporates the following: (i) decomposition of memory-intensive lookup tables
(LUTs) into compact LUTs that fit in GPU SRAM, (ii) a two-phase kernel for
coordinating thread read/write positions using lightweight auxiliary variables,
and (iii) transformer-block-level decompression to minimize latency.
Experiments on recent models, including Llama-3.1, Qwen-2.5, and Gemma-3,
validates our hypothesis that DFloat11 achieves around 30% model size reduction
while preserving bit-for-bit exact outputs. Compared to a potential alternative
of offloading parts of an uncompressed model to the CPU to meet memory
constraints, DFloat11 achieves 1.9-38.8x higher throughput in token generation.
With a fixed GPU memory budget, DFloat11 enables 5.3-13.17x longer context
lengths than uncompressed models. Notably, our method enables lossless
inference of Llama-3.1-405B, an 810GB model, on a single node equipped with
8x80GB GPUs. Our code and models are available at
https://github.com/LeanModels/DFloat11.
6801228e8b6f40490fb179f8
Tianyi Zhang
claimed_verified
2025-04-18T09:39:45.219Z
66a3196b98016fc5e1731bfb
https://cdn-avatars.huggingface.co/v1/production/uploads/66a3196b98016fc5e1731bfb/Kx3IdfxIXOl4U-ox7CoxT.png
Tianyi Zhang
user
LeanQuant
6801228e8b6f40490fb179f9
Yang Sui
claimed_verified
2025-04-20T15:03:33.167Z
63787b13500186f250ba377c
/avatars/96bb6051662109e9cd25e6df7d738f17.svg
yangsui
user
yangsui
6801228e8b6f40490fb179fa
Shaochen Zhong
6801228e8b6f40490fb179fb
Vipin Chaudhary
6801228e8b6f40490fb179fc
Xia Hu
6801228e8b6f40490fb179fd
Anshumali Shrivastava
70% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU
  Inference via Dynamic-Length Float
Large Language Models (LLMs) have grown rapidly in size, creating significant
challenges for efficient deployment on resource-constrained hardware. In this
paper, we introduce Dynamic-Length Float (DFloat11), a lossless compression
framework that reduces LLM size by 30% while preserving outputs that are
bit-for-bit identical to the original model. DFloat11 is motivated by the low
entropy in the BFloat16 weight representation of LLMs, which reveals
significant inefficiency in existing storage format. By applying entropy
coding, DFloat11 assigns dynamic-length encodings to weights based on
frequency, achieving near information-optimal compression without any loss of
precision. To facilitate efficient inference with dynamic-length encodings, we
develop a custom GPU kernel for fast online decompression. Our design
incorporates the following: (i) decomposition of memory-intensive lookup tables
(LUTs) into compact LUTs that fit in GPU SRAM, (ii) a two-phase kernel for
coordinating thread read/write positions using lightweight auxiliary variables,
and (iii) transformer-block-level decompression to minimize latency.
Experiments on recent models, including Llama-3.1, Qwen-2.5, and Gemma-3,
validates our hypothesis that DFloat11 achieves around 30% model size reduction
while preserving bit-for-bit exact outputs. Compared to a potential alternative
of offloading parts of an uncompressed model to the CPU to meet memory
constraints, DFloat11 achieves 1.9-38.8x higher throughput in token generation.
With a fixed GPU memory budget, DFloat11 enables 5.3-13.17x longer context
lengths than uncompressed models. Notably, our method enables lossless
inference of Llama-3.1-405B, an 810GB model, on a single node equipped with
8x80GB GPUs. Our code and models are available at
https://github.com/LeanModels/DFloat11.
DFloat11, a lossless compression framework, reduces Large Language Model sizes by 30% through entropy coding, enabling efficient deployment and significantly higher throughput on resource-constrained hardware.
Dynamic-Length Float DFloat11 LLM Large Language Models lossless compression entropy coding BFloat16 weights GPU kernel memory-intensive lookup tables LUTs GPU SRAM transformer-block-level decompression token generation context lengths