Effective Interplay between Sparsity and Quantization: From Theory to Practice
Simla Burcu Harma Ayan Chakraborty Elizaveta Kostenok Danila Mishin Dongho Ha Babak Falsafi Martin Jaggi Ming Liu Yunho Oh Suvinay Subramanian Amir Yazdanbakhsh
The increasing size of deep neural networks (DNNs) necessitates effective model compression to reduce their computational and memory footprints. Sparsity and quantization are two prominent compression methods that have been shown to reduce DNNs' computational and memory footprints significantly while preserving model accuracy. However, how these two methods interact when combined together remains a key question for developers, as many tacitly assume that they are orthogonal, meaning that their combined use does not introduce additional errors beyond those introduced by each method independently. In this paper, we provide the first mathematical proof that sparsity and quantization are non-orthogonal. We corroborate these results with experiments spanning a range of large language models, including the OPT and LLaMA model families (with 125M to 8B parameters), and vision models like ViT and ResNet. We show that the order in which we apply these methods matters because applying quantization before sparsity may disrupt the relative importance of tensor elements, which may inadvertently remove significant elements from a tensor. More importantly, we show that even if applied in the correct order, the compounded errors from sparsity and quantization can significantly harm accuracy. Our findings extend to the efficient deployment of large models in resource-constrained compute platforms to reduce serving cost, offering insights into best practices for applying these compression methods to maximize hardware resource efficiency without compromising accuracy.
theory of compression, model compression, quantization, max-scaled numerical encoding, sparsity, unstructured sparsity, structured sparsity, N:M sparsity, large language models, magnitude pruning, post-training quantization, efficient inference
We mathematically analyze the interplay of sparsity and quantization, proving they are not orthogonal operations. This means their combined error is greater than the sum of their parts, especially due to quantization.