MCNC: Manifold Constrained Network Compression
<|reference_start|>MCNC: Manifold Constrained Network Compression: The outstanding performance of large foundational models across diverse tasks-from computer vision to speech and natural language processing-has significantly increased their demand. However, storing and transmitting these models pose significant challenges due to their massive size (e.g., 350GB for GPT-3). Recent literature has focused on compressing the original weights or reducing the number of parameters required for fine-tuning these models. These compression methods typically involve constraining the parameter space, for example, through low-rank reparametrization (e.g., LoRA) or quantization (e.g., QLoRA) during model training. In this paper, we present MCNC as a novel model compression method that constrains the parameter space to low-dimensional pre-defined and frozen nonlinear manifolds, which effectively cover this space. Given the prevalence of good solutions in over-parameterized deep neural networks, we show that by constraining the parameter space to our proposed manifold, we can identify high-quality solutions while achieving unprecedented compression rates across a wide variety of tasks. Through extensive experiments in computer vision and natural language processing tasks, we demonstrate that our method, MCNC, significantly outperforms state-of-the-art baselines in terms of compression, accuracy, and/or model reconstruction time.<|reference_end|>
@article{thrash2024mcnc:,
    title={MCNC: Manifold Constrained Network Compression},
    author={Chayne Thrash, Ali Abbasi, Parsa Nooralinejad, Soroush Abbasi
  Koohpayegani, Reed Andreas, Hamed Pirsiavash, Soheil Kolouri},
    journal={arXiv preprint arXiv:2406.19301},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.19301},
    primaryClass={cs.LG}
}