This paper proposes a Pruning in Training (PiT) framework of learning to reduce the parameter size of networks. Different from existing works, our PiT framework employs the sparse penalties to train networks and thus help rank the importance of weights and filters. Our PiT algorithms can directly prune the network without any fine-tuning. The pruned networks can still achieve comparable performance to the original networks. In particular, we introduce the (Group) Lasso-type Penalty (L-P /GL-P), and can alternatively implement them using (Group) Split LBI (S-P / GS-P)-an regularization solution path with corresponding penalties to regularize the networks, and a pruning strategy proposed by Fu et al. (2016b)  is used to help prune the network. We conduct the extensive experiments on MNIST, Cifar-10, and miniImageNet. The results validate the efficacy of our proposed methods. Remarkably, on MNIST dataset, our PiT framework can produce a small network which only has 17.5% parameter size of LeNet-5, and achieves the 98.47% recognition accuracy.
This paper introduces an approach to pruning while training a network. This is interesting and experiments show interesting results in several datasets including ResNet18
Here are a few comments:
 - Pruning or regularization for compression is not new. Alvarez and Wen have used group lasso types as suggested in the paper and some others such as Alvarez and Salzmann (Compression aware training NIPS 2017) and Wen (Coordinating filters ICCV2017) have used low-rank type of while training. How is this different to those? They also do not need any sort of fine tuning and more importantly, they show this can scale to large networks and datasets. 
- These last two works I mentioned promote redundancy, similarly to what is suggested in the paper. Would be good to get them cited and compared. Important from those is the training methodology to avoid relevant overheads. How is that happening in the current approach
- While I like the approach, would be nice to see how this scale.  All for methods above (and others related) do work on full imagenet to show performance.  For ResNet, cleaning the network is not really trivial (near the block), is that a limitation?
- Why limiting experiments to small networks and datasets? Time wise, how does this impact the training time?
- Why limiting the experiments to at most 4 layers? 
- I am certainly not impressed by results on fully connected layers in MNIST. While the experiment is interesting does not seem to be of value as most networks do not have those layers anymore.
- Main properties of this approach are selecting right filters while training without compromising accuracy or needing fine tuning. While that is of interest, i do not see the difference with other related works (such as those I cited above)
- As there is enough space, I would like to see top-1 results for comprehensive comparison. 
- I think tables need better captions for being self-contained. I do not really understand what i see in table 5 for instance. 
- Droping 13% of top5 accuracy does not seem negligible, what is the purpose there? Would also be interesting to compare then with any other network with that performance. 
- What about flops and forward time? Does this pruning strategy help there?