DYNAMIC SPARSE GRAPH FOR EFFICIENT DEEP LEARNING
We propose to execute deep neural networks (DNNs) with dynamic and sparse graph (DSG) structure for compressive memory and accelerative execution during both training and inference. The great success of DNNs motivates the pursuing of lightweight models for the deployment onto embedded devices. However, most of the previous studies optimize for inference while neglect training or even complicate it. Training is far more intractable, since (i) the neurons dominate the memory cost rather than the weights in inference; (ii) the dynamic activation makes previous sparse acceleration via one-off optimization on fixed weight invalid; (iii) batch normalization (BN) is critical for maintaining accuracy while its activation reorganization damages the sparsity. To address these issues, DSG activates only a small amount of neurons with high selectivity at each iteration via a dimensionreduction search and obtains the BN compatibility via a double-mask selection. Experiments show significant memory saving (1.7-4.5x) and operation reduction (2.3-4.4x) with little accuracy loss on various benchmarks.
[Overview]
In this paper, the authors proposed to use dynamic sparse computation graph for reducing the computation memory and time cost in deep neural network (DNN). This method is applicable in both DNN training and inference. Unlike most of previous work that focusing on the reduction of computation during the inference time, this new method propose a dynamic computation graph by pruning the activations on the fly during the training of inference, which is an interesting and novel exploration. In the experiments, the authors performed extensive experiments to demonstrate the effectiveness of the proposed method compared with several baseline methods and original models. It is clear to me that this method helps to reduce the memory cost and computation cost for both DNN training and inference.
[Strengthes]
1. This paper addresses the computational burden in both memory and time from a novel angle than previous network pruning methods. It can be applied to reduce the computation in both network training and inference, but also preserve the representation ability of the network.
2. To endow the network compression in training and inference, the authors proposed to mute the low-activated neurons so that the computations merely happened on those selected neurons. 
3. For the selection, the authors proposed a simple but efficient dimension reduction methods, random sparse projection, to project the original activations and weights into a lower-dimensional space and compute the approximated response map in such a lower dimension space, which the selection is based on.
4. The authors performed comprehensive experiments to demonstrate the effectiveness the proposed method for network compression. Those results are insightful and solid.
[Questions]
1. Is the sparsity of each layer the same across the whole network? It would be nice if the authors could perform some ablation studies on varied sparsity in different layers, maybe just with some heuristic methods, e.g., decreasing the sparsity from lower layer to upper layers. As the authors mentioned, higher sparsity causes a larger degradation on deeper network. I am curious that whether there are some better way to set the sparsity.
2. During the training of the network, how the activation evolve? It would be interesting to show how the selected activation changes across the training time for the same training sample.  This might provide some insights on when the activations begin to converge to a stable state, and how it varies layer by layer. 
3. Following the above questions, is there any stage that the sparsity can be fixed without further computation for selection. In generally, the training proceeds for a number of epochs. It would be nice if we can observe some convergence on the selected activations and then we can suspend the selection for saving the computation burden.
[Conclusion]
This paper present an interesting and novel approach for network pruning in both training and inference. Unlike most of the previous work, it pruning the activations in each layer though a dimension reduction strategy. From the experiments, this method achieved an obvious improvement for reducing the computation memory and time cost in training and inference stages. I think this paper has prompted a new direction of efficient deep neural network.