Convex Distillation: Efficient Compression of Deep Networks via Convex Optimization
<|reference_start|>Convex Distillation: Efficient Compression of Deep Networks via Convex Optimization: Deploying large and complex deep neural networks on resource-constrained edge devices poses significant challenges due to their computational demands and the complexities of non-convex optimization. Traditional compression methods such as distillation and pruning often retain non-convexity that complicates fine-tuning in real-time on such devices. Moreover, these methods often necessitate extensive end-to-end network fine-tuning after compression to preserve model performance, which is not only time-consuming but also requires fully annotated datasets, thus potentially negating the benefits of efficient network compression. In this paper, we introduce a novel distillation technique that efficiently compresses the model via convex optimization -- eliminating intermediate non-convex activation functions and using only intermediate activations from the original model. Our approach enables distillation in a label-free data setting and achieves performance comparable to the original model without requiring any post-compression fine-tuning. We demonstrate the effectiveness of our method for image classification models on multiple standard datasets, and further show that in the data limited regime, our method can outperform standard non-convex distillation approaches. Our method promises significant advantages for deploying high-efficiency, low-footprint models on edge devices, making it a practical choice for real-world applications. We show that convex neural networks, when provided with rich feature representations from a large pre-trained non-convex model, can achieve performance comparable to their non-convex counterparts, opening up avenues for future research at the intersection of convex optimization and deep learning.<|reference_end|>
@article{varshney2024convex,
    title={Convex Distillation: Efficient Compression of Deep Networks via Convex
  Optimization},
    author={Prateek Varshney and Mert Pilanci},
    journal={arXiv preprint arXiv:2410.06567},
    year={2024},
    archivePrefix={arXiv},
    eprint={2410.06567},
    primaryClass={cs.LG}
}