NEPENTHE: Entropy-Based Pruning as a Neural Network Depth's Reducer
<|reference_start|>NEPENTHE: Entropy-Based Pruning as a Neural Network Depth's Reducer: While deep neural networks are highly effective at solving complex tasks, their computational demands can hinder their usefulness in real-time applications and with limited-resources systems. Besides, for many tasks it is known that these models are over-parametrized: neoteric works have broadly focused on reducing the width of these networks, rather than their depth. In this paper, we aim to reduce the depth of over-parametrized deep neural networks: we propose an eNtropy-basEd Pruning as a nEural Network depTH's rEducer (NEPENTHE) to alleviate deep neural networks' computational burden. Based on our theoretical finding, NEPENTHE focuses on un-structurally pruning connections in layers with low entropy to remove them entirely. We validate our approach on popular architectures such as MobileNet and Swin-T, showing that when encountering an over-parametrization regime, it can effectively linearize some layers (hence reducing the model's depth) with little to no performance loss. The code will be publicly available upon acceptance of the article.<|reference_end|>
@article{liao2024nepenthe:,
    title={NEPENTHE: Entropy-Based Pruning as a Neural Network Depth's Reducer},
    author={Zhu Liao, Victor Qu'etu, Van-Tam Nguyen, Enzo Tartaglione},
    journal={arXiv preprint arXiv:2404.16890},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.16890},
    primaryClass={cs.LG cs.AI}
}