MCNC: Manifold-Constrained Reparameterization for Neural Compression
Chayne Thrash Reed Andreas Ali Abbasi Parsa Nooralinejad Soroush Abbasi Koohpayegani Hamed Pirsiavash Soheil Kolouri
The outstanding performance of large foundational models across diverse tasks,
from computer vision to speech and natural language processing, has significantly
increased their demand. However, storing and transmitting these models poses
significant challenges due to their massive size (e.g., 750GB for Llama 3.1 405B).
Recent literature has focused on compressing the original weights or reducing the
number of parameters required for fine-tuning these models. These compression
methods generally constrain the parameter space, for example, through low-rank
reparametrization (e.g., LoRA), pruning, or quantization (e.g., QLoRA) during
or after the model training. In this paper, we present a novel model compres-
sion method, which we term Manifold-Constrained Neural Compression (MCNC).
This method constrains the parameter space to low-dimensional pre-defined and
frozen nonlinear manifolds, which effectively cover this space. Given the preva-
lence of good solutions in over-parameterized deep neural networks, we show that
by constraining the parameter space to our proposed manifold, we can identify
high-quality solutions while achieving unprecedented compression rates across
a wide variety of tasks and architectures. Through extensive experiments in
computer vision and natural language processing tasks, we demonstrate that our
method significantly outperforms state-of-the-art baselines in terms of compres-
sion, accuracy, and/or model reconstruction time. Our code is publicly available at
https://github.com/mint-vu/MCNC.
Model Compression, LoRA, PEFT, Transformers, ViT
A method that compresses large models by limiting parameters to low-dimensional manifolds. It achieves superior compression of CNNs, ViTs, and LLMs across tasks compared to current techniques.