Deep neural networks have demonstrated unprecedented success in various knowledge management applications. However, the networks created are often very complex, with large numbers of trainable edges which require extensive computational resources. We note that many successful networks nevertheless often contain large numbers of redundant edges. Moreover, many of these edges may have negligible contributions towards the overall network performance. In this paper, we propose a novel iSparse framework, and experimentally show, that we can sparsify the network, by 30-50%, without impacting the network performance. iSparse leverages a novel edge significance score, E, to determine the importance of an edge with respect to the final network output. Furthermore, iSparse can be applied both while training a model or on top of a pre-trained model, making it a retraining-free approach -leading to a minimal computational overhead. Comparisons of iSparse against PFEC, NISP, DropConnect, and Retraining-Free on benchmark datasets show that iSparse leads to effective network sparsifications.
The paper proposed a method for network sparsification based on the significance of each edge (weight). Unlike some of the existing works, edge significance in this work is explicitly defined based on their influence over the network output.
The algorithm can be summarized as follows:
1. Compute the significance of each activation (neuron) using an existing method (infinite feature selection).
2. Compute the significance of each edge as the product between the neuron significance and the absolute edge weight.
3. Sort the edges according to their significance scores and keep the top portion.
I'm mainly concerned about the limited technical novelty: the proposed technique is essentially a heuristic. Specifically, it is not clear why the significance of each edge should be defined according to equation (5) rather than some other forms. The intuition behind masking out the gradients of in-significant edges in (10), i.e., "we argue that any edge that does not contribute towards the final model output, must not be included in the back-propagation", is again a heuristic that lacks justification. If theoretical analysis is not possible, it might be necessary for the authors to conduct controlled experiments/ablation studies to show that some of the design choices made in the paper are indeed superior over other alternatives.
Another of my concern is that, if the goal is to make the sparsification decision aware of the network output (e.g., the value of the loss function), a simpler approach would be to enforce L1 regularization over the edges. This way, edges that do not lead to significant impact on the loss would be automatically pruned away. I wonder how would the proposed approach compare against this simple baseline.
Other suggestions:
* Empirical evaluation is conducted using LeNet and VGG16. It would be interesting to extend the analysis to some other seminal architectures, such as ResNet, Inception and MobileNets.
* It would be informative to report the hardware configuration used to obtain the execution time in Figure 5. Note the relative inference cost of different models may differ substantially over different hardware platforms.
* Writing of the paper can probably be polished further for better clarity.