In this paper, we propose a framework, called MaskConvNet, for ConvNets filter pruning. MaskConvNet provides elegant support for training budget-aware pruned networks from scratch, by adding a simple mask module to a ConvNet architecture. MaskConvNet enjoys several advantages -(1) Flexible, the mask module can be integrated with any ConvNets in a plug-and-play manner. (2) Simple, the mask module is implemented by a hard Sigmoid function with small number of trainable mask variables, adding negligible memory and computational overheads to the networks during training. (3) Effective, it is able to achieve competitive pruning rate while maintaining comparable accuracy with the baseline ConvNets without pruning, regardless of the datasets and ConvNet architectures used. (4) Fast, it is observed that the number of training epochs required by MaskConvNet is close to training a baseline without pruning. (5) Budget-aware, with a sparsity budget on target metric (e.g. model size and FLOP), MaskConvNet is able to train in a way that the optimizer can adaptively sparsify the network and automatically maintain sparsity level, till the pruned network produces good accuracy and fulfill the budget constraint simultaneously. Results on CIFAR-10 and ImageNet with several ConvNet architectures show that MaskConvNet works competitively well compared to previous pruning methods, with budgetconstraint well respected. Code is available at https://www.dropbox.com/ s/c4zi3n7h1bexl12/maskconv-iclr-code.zip?dl=0. We hope MaskConvNet, as a simple and general pruning framework, can address the gaps in existing literate and advance future studies to push the boundaries of neural network pruning.
This paper proposes a masking process to improve the pruning of DNN. In addition, the algorithm proposes to automatically allocate the pruning rates over layers. 
On the positive side:
- I do believe the main contribution is automatically allocating the pruning rates over the network. 

Related works:
- How does this relate to methods using gating to prune in the presence of residual layers or BN?
- Paper claims multiple times that related works need a train-prune-retrain process which is only valid for post-processing works. 
- I missed sparsity promoting works related to training from scratch where similar masks are implicitly used during training (even not explicitly stated) to maintain the zeros in the network. At least in [1,2] the training time is the same as the time used for training a network from scratch (same claim as in this paper). 
Method:
- The mask is trained using a sigmoid function and claims this will be representative of the relevance of the 'neuron' within the layer. How is this really related?
- Why the initialization of the mask is to the mean? I think I am confused there. For initialization, I would argue all the neurons/parameters are relevant, right?
- The claim that this method is 'much simpler' is a bit subjective. I do not see why. Please elaborate. 
- I am not sure if the claim of pruning filters and not considering the bn layer is correct. I would tend to think that, if pruning a neuron, the corresponding BN module should be modified (that is, propagating the zero to subsequent layers). 
- I do like the extension to residual connections. It would be great to have more and clearer details on how is this done. Would this also apply to the UNET type of architecture?
- What is the intuition behind Eq 4 and how it is related to the relevance of a parameter? 
- The extension to multiple metrics is of interest, however, there is little detailed there. How is the automatic allocation done? This is repeated in section 3.3 but details are missing. 
- My understanding is that the regularization multiplier is affected by the learning rate, therefore their effect is lower as the training progresses. In this case, seems the opposite, right? (page 6 before 3.4). 
- The part with the sparsity budget is interesting. What guarantees are that the newly enabled neurons are actually useful? Could it be possible that the budget suggested is not the right for the task at hand and, therefore, the additional parameters are not really relevant / needed? 

Experiments:
- There is loss with little sparsity when it comes to Imagenet (15 and 17% pruning) does not seem very promising even the training time is similar. 
- Seems like the experiments and comparisons to L1-pruning are not very surprising. It would be nice to have more comprehensive numbers. For imagenet, if using Resnet-50, would be easier to compare to other numbers. 
- In the imagenet comparison, L1-pruning is the version-A of the paper. What about the others or why that particular model?
- How are the actual groups made? 
Minor details:
- Please improve figure 1. It is not easy to understand. Same with Figure 3. What is seen in Figure 4. 
- I do believe the WideResNet-28-10 number of parameters for BAR is not correct. 
- Section 4.2 is a bit overselling. I do not see 'much-higher' parameter sparsity. The claims are mostly valid for VGG type of networks in this particular setting. 

[1] Learning the Number of Neurons in Deep Networks, NeurIps 2016
[2] Compression-aware training of DNN, NeurIps 2017