We propose a new approach, known as the iterative regularized dual averaging (iRDA), to improve the efficiency of convolutional neural networks (CNN) by significantly reducing the redundancy of the model without reducing its accuracy. The method has been tested for various data sets, and proven to be significantly more efficient than most existing compressing techniques in the deep learning literature. For many popular data sets such as MNIST and CIFAR-10, more than 95% of the weights can be zeroed out without losing accuracy. In particular, we are able to make ResNet18 with 95% sparsity to have an accuracy that is comparable to that of a much larger model ResNet50 with the best 60% sparsity as reported in the literature.
iRDA Method for sparse convolutional neural networks 
This paper considers the problem of training a sparse neural network. The main motivation is that usually all state of the art neural network’s size or the number of weights is enormous and saving them in memory is costly. So it would be of great interest to train a sparse neural network. To do so, this paper proposed adding l1 regularizer to RDA method in order to encourage sparsity throughout training. Furthermore, they add an extra phase to  RAD algorithm where they set the stochastic gradient of zero weights to be zero. They show experimentally that the method could give up to 95% sparsity while keeping the accuracy at an acceptable level. 
More detail comments: 
1- In your analysis for the convergence, you totally ignored the second step. How do you show that with the second step still the method converge? 
2- \bar{w} which is used in the thm 1, is not introduced. 
3- In eq 5, you say g_t is subfunction. What is it? 
4- When does the algorithm switch from step 1 to step 2? 
5- In eq 35 what is \sigma? 
6- What is the relation between eq 23 and 24? The paper says 23 is an approximation for 24 but the result of 23 is a point and 24 is a function. 
7- What is MRDA in the Fig 1?