Recent DNN pruning algorithms have succeeded in reducing the number of parameters in fully connected layers, often with little or no drop in classification accuracy. However, most of the existing pruning schemes either have to be applied during training or require a costly retraining procedure after pruning to regain classification accuracy. In this paper we propose a cheap pruning algorithm based on difference of convex functions (DC) optimisation. We also provide theoretical analysis for the growth in the Generalization Error (GE) of the new pruned network. Our method can be used with any convex regulariser and allows for a controlled degradation in classification accuracy, while being orders of magnitude faster than competing approaches. Experiments on common feedforward neural networks show that for sparsity levels above 90% our method achieves 10% higher classification accuracy compared to Hard Thresholding.
This paper casts the pruning optimization problem of NetTrim as a difference of convex problems, and uses DCA to obtain the smaller weight matrix; this algorithm is also analyzed theoretically to provide a bound on the generalization error of the pruned network.
However, there are many questions that aren't answered in the paper that make it difficult to evaluate: in particular, some experimental results leave open more questions for performance analysis. 
Quality: of good quality, but incomplete.
Clarity: clear with some typos
Originality: a new approach to the NetTrim algorithm, which is somewhat original, and a new generalization bound for the algorithm.
Significance: somewhat significant.
PROS
- A very efficient algorithm for pruning, which can run orders of magnitude faster than the approaches that were compared to on certain architectures.
- An interesting generalization bound for the pruned network which is in line experimentally with decreasing robustness to pruning on layers close to the input.
CONS
- Non-trivial loss of accuracy on the pruned network, which cannot be estimated for larger-scale pruning as the experiments only prune one layer.
- No in-depth analysis of the generalization bound.
Main questions:
- You mention you use a variant of DCA: could you detail what differences Alg. 2 has with classical DCA?
- Where do you use the 0-1 loss in Thm. 3.2?
- I think your result in Theorem 3.2 would be significantly stronger if you could provide an analysis of the bound you obtain: in which cases can we expect certain terms to be larger or smaller, etc.
- Your experiments in section 4.2 show a non-trivial degradation of the accuracy with FeTa. Although the time savings seem worth the tradeoff to prune *one* layer, have you run the same experiments when pruning multiple layers? Could you comment on how the accuracy evolves with multiple pruned layers?
- It would be nice to see the curves for NetTrim and/or LOBS in Fig. 2.
- Have you tried retraining the network after pruning? Did you observe the same behavior as mentioned in (Dong et al., 2017) and (Wolfe et al., 2017)? 
- It would be interseting to plot the theoretical (pessimistic) GE bound as well as the experimental accuracy degradation. 
Nitpicks:
-Ubiquitous (first paragraph)
-difference of convex problemS
- The references should be placed before the appendix.
- The amount of white space should be reduced (e.g. around Eq. (1)).