Deep neural networks (DNNs) can be huge in size, requiring a considerable amount of energy and computational resources to operate, which limits their applications in numerous scenarios. It is thus of interest to compress DNNs while maintaining their performance levels. We here propose a probabilistic importance inference approach for pruning DNNs. Specifically, we test the significance of the relevance of a connection in a DNN to the DNN's outputs using a nonparemetric scoring test and keep only those significant ones. Experimental results show that the proposed approach achieves better lossless compression rates than existing techniques.
This paper introduces a nonparametric score test to estimate the importance of network connections on the final output of Deep Neural Networks. They derive this by modeling the log-transformed joint density of each connection and final output in a tensor product reproducing kernel Hilbert space. They finally derive an asymptotic distribution of the proposed test statistics which only depends on the eigenvalues of the kernel. This importance test is applied to ranking the importance of each connection in Multilayer Perceptron Networks, and Convolutional Networks and sparsifying the networks by removing the least important connections. The method is applied post-training to fully trained networks but evaluated by retraining the sparsified network from scratch. The method is demonstrated in experiments on compressing three networks. 
I believe this paper is a borderline accept. It provides a more statistically principled method to examine the importance of connections in a Neural Network and rank them compared to existing compression methods. Due to this ranking, there is a clear method to easily adjust the target compression rate. They are able to achieve high lossless compression rates. However, the benefits shown in the empirical results could be more convincing. It lacks baselines on more complex networks and could benefit from more empirical analysis of the theoretical benefits and properties of this approach.
Pros: 
The method is able to maintain accuracy while achieving high compression rates on MNIST and CIFAR10. It does better than the baselines compared to the small networks. They show a capability for increasing generalizability by decreasing error rates. 
It provides a well derived statistically principled method to examine the importance of connections and rank them.
Indicates an additional use to visualize the importance of features.
Cons:
The empirical results could be clearer. It lacks baselines for larger models on Cifar10. Could you compare it with the published results of other algorithms? How does it do on larger networks like Imagenet?
Computational efficiency is mentioned but could be examined in greater detail.  How long does it take to run and how is that affected by model size?
The experimental setting is somewhat unclear. The baseline Louizos et al. (2017)  was designed to optimize group sparsity/speed, but the experimental results here only examine the compression rate. Was the baseline run to optimize speed or sparsity?
It would be interesting to examine the correlation of importance score with actual impact on network performance. This might be done with a comparison with random pruning or pruning higher importance connections. It might be useful to examine the performance of the networks after pruning nodes of differing importance without full retraining or just fine-tuning. It is unclear how important full retraining is in this method.
It would be interesting to visualize the importance of features at different depths in the deep convolutional networks.
Minor suggestions: In the introduction, you mention l0 and l1 norm methods, but cite Han et al. (2015) which compares l1 and l2 norm and found l2 norm to be better overall.
typos:
In Abstract: nonparemtric scoring test  -> nonparametric scoring test  
In 4.4: sample averarge -> sample average
In 4.5: ASYMPTOTICALLY DISTRIBUTION  ->   ASYMPTOTIC DISTRIBUTION