This paper studies structured sparse training of CNNs with a gradual pruning technique that leads to fixed, sparse weight matrices after a set number of epochs. We simplify the structure of the enforced sparsity so that it reduces overhead caused by regularization. The proposed training methodology explores several options for structured sparsity.
We study various tradeoffs with respect to pruning duration, learning-rate configuration, and the total length of training. We show that our method creates a sparse version of ResNet50 and ResNet50v1.5 on full ImageNet while remaining within a negligible <1% margin of accuracy loss. To ensure that this type of sparse training does not harm the robustness of the network, we also demonstrate how the network behaves in the presence of adversarial attacks. Our results show that with 70% target sparsity, over 75% top-1 accuracy is achievable.
SUMMARY
-------
This paper explores a series of incremental variations of existing pruning techniques for compressing Resnet-50 for ImageNet. Specifically, it proposes concentrating all pruning during an early "era" of training (the first 20-50 epochs out of 100 total). It also explores hybrids between sparse pruning and structured pruning. Finally, it considers the adversarial robustness of the resulting networks to the FGSM attack. 
This paper makes no novel proposals and experiments are minimal. There are no clear takeaways from the results of these experiments. The goals of the paper are unclear, and it is difficult to compare this paper to existing work. 
This paper has no clear motivation and makes no tangible contributions to the literature and, therefore, I recommend a rejection. 
CONTRIBUTIONS
-------------
1) A study of the appropriate window ("pruning era") for pruning Resnet-50 on ImageNet and TinyImageNet
2) A study of the tradeoffs between various forms of structured and unstructured pruning.
3) An analysis of the adversarial robustness of the pruned networks.
DETAILED COMMENTS
------
PROBLEMS ADDRESSED
It was challenging to discern the specific problems that this paper sought to address and, relatedly, the goals that the paper sought to achieve. The introduction of the paper lists a wide variety of problems in the existing literature:
1) Paragraph 3: Structured sparsity introduces "regularization and computational overhead."
2) Paragraph 3: "Coarse-grained sparsity" cannot eliminate enough parameters to perform well on "edge devices."
3) Paragraph 4: Dynamic sparsity techniques require more training epochs.
4) Paragraph 4: Dynamic sparsity techniques do not preserve network accuracy (1-2 percentage point drop at 80% sparsity).
5) Paragraph 4: Dynamic sparsity requires reconfiguring the sparsity pattern frequently, which is computationally expensive.
The paper does not justify the fact that any of these are actually problems, nor does it make any attempt to quantify the extent of these problems. Moreover, the proposed techniques do not resolve any of these problems. Corresponding to the numbers above:
1) The paper never measures this overhead nor justifies that it is a problem in practice. Meanwhile, the techniques proposed in the paper introduce substantial overhad of their own, including training for an extra ten epochs. It is possible that the techniques proposed in this paper have worse overhead than the techniques that are criticized in the introduction. Since the paper provides on numbers either way, it is impossible to tell. In short, computational cost is a key part of the author's argument despite the fact that there is no empirical support for any of these claims.
2) I believe the paper means that, in order to get to sufficient levels of sparsity to work on "edge devices," accuracy drops unacceptably far. What does the paper mean by "edge devices," what are sufficient levels of sparsity, and what does it mean for accuracy to drop unacceptably far? The paper has numbers for the proposed methods, so it should be possible to make this comparison if such baselines are explicit.
3) The proposed techniques also require the same number of additional training epochs, so this complaint is unaddressed.
4) The proposed techniques show a 2-3 percentage point drop at 80% sparsity (Table 1), which is actually worse than the technique that the authors criticize.
5) The proposed techniques require pruning after every single training step during the "pruning era." This is likely to be more computationally expensive than any of the other gradual pruning and dynamic sparsity techniques listed, which prune at intervals of hundreds or thousands of iterations. In addition, the authors never justify why changing the sparsity pattern frequently throughout training will affect performance. On GPUs with modern frameworks, I see no reason why this should matter so long as the sparsity pattern does not change too frequently (although that is exactly what this paper proposes to do during the "pruning era").
GOALS
It was also challenging to discern the goals of the paper. Was it:
1) To produce the smallest possible trained networks with the highest possible accuracy?
2) To reduce the cost of obtaining a pruned network for inference-time? (Or to reduce the cost of obtaining a sufficiently efficient pruned network for inference-time?)
3) To reduce the cost of training neural networks in general by pruning them during training?
In the introduction and the related work section, these goals go unstated, making it difficult to determine how this paper compares to existing work. The comparisons provided in the paper focus on specific aspects of each related work rather than the entire picture. For example, in comparison to Mao et al., the authors claim better accuracy at one sparsity level, implying goal 1. However, for to Lym et al., the paper focuses on the computational costs of training the network, implying goal 3.
UNJUSTIFIED CLAIMS ABOUT NEURAL NETWORK COMPUTATION
Throughout the paper, there are a number of unjustified claims about which neural network configurations will perform better on contemporary hardware. Considering computational efficiency appears to be a key element of the paper's argument, these claims require citations or - particularly when various configurations are compared to one another - empirical support. Some examples:
* Section 1, Paragraph 3: "The regularization term [of structured sparsity] modifies the original training and can be expensive in hardware."
* Section 1, Paragraph 3: "The final network [from Lym et al. 2019] contains an insufficient degree of sparsity for deployment on edge devices."
* Section 1, Paragraph 4: "Continuous reconfiguration of the sparsity pattern is expensive as it does not allow for compression of weights during training"
* Section 1, Paragraph 5: "having a fixed sparse multiply-accumulate pattern allows weight compression during training and can save compute and energy in hardware"
* Section 5, Paragraph 2: "A strict parameter allows the hardware mapping to plan for a fixed number of multiply-accumulate operations."
* Section 5, Paragraph 2: "Regularization, although useful in forcing the network to learn prunable weights, adds more irregularity to computation flow."
PRUNING TECHNIQUES
* Recomputing the pruning mask at every training step seems gratuitously inefficient.
* Sorting the weights in the entire network shouldn't be particularly inefficient if it isn't done on every single iteration. (Section 2.1 paragraph 1)
* Why do you maintain the same number of weights in each convolutional filter with window pruning? (Presumably for performance reasons, but you never say that.)
* None of the pruning methods are novel. They're simply various permutations of structured and unstructured magnitude pruning as proposed by many others in the literature.
EXPERIMENTS
* Section 3.1 Paragraph 2: It appears that you are exploring the best "pruning era." If you are to do so, you will have to sweep over (1) the length of the pruning era (2) the starting epoch of the pruning era, and (3) the shape of the function used to determine sparsity. Instead, it sounds like you tried two arbitrary pruning eras (0-30 and 30-50). Likewise, in Paragraph 3, you test only a small number of possible scenarios.
* Section 3.1 is generally hard to parse. It is unclear what you are studying. The ideal pruning era? The relative performance of the pruning methods introduced in section 2? 
* How many times did you replicate each experiment? You should ideally include at least 3 (and preferrably 5) replicates with mean and stddev reported.
* What baselines are you including? You should include a random pruning baseline and you should ideally replicate any methods that you compare to.
RESULTS
* Section 4.1: The data you refer to is in an appendix even though it is crucial to the main body of the paper. The appendices should contain material that is nonessential for making sense of the paper.
* Section 4.2 Paragraph 1: Are these numbers good? A standard sparse pruning technique (Gale et al. 2019, https://arxiv.org/pdf/1902.09574.pdf) achieve 70% sparsity without any change in accuracy. Please include baselines comparing to other methods in the literature.
* Table 2: It is difficult to compare the results in these papers. PruneTrain aims to reduce the cost of training and measures cost reductions in FLOPS. If you intend to compare against this paper, you should quantify the cost of training using your method against that of PruneTrain. Merely presenting sparsity and accuracy numbers is insufficient. Likewise for the dynamic sparsity experiments. What is your goal in showing this comparison, and did Mostafa and Wang share that goal when they justified their technique?
* You do not describe the hyperparameters for Intraepoch pruning (the balance between window and CK - last paragraph of 2.1.1)
ADVERSARIAL ROBUSTNESS
Considering the fact that this paper focuses on proposing new variations of existing pruning techniques, any discussion of adversarial robustness seems to be (1) out of place and (2) an afterthought. If the authors delete a half-page of content (one phrase from the abstract, a paragraph and bullet from the introduction, and a paragraph each from sections 3 and 4), this content could be removed with minimal impact to the paper's main contributions. The content on adversarial robustness is cursory, uses a weak and out-of-date attack (FGSM), and does not compare to any other pruning methods. In fact, the one comparison is to the results in a paper (Wang et al, 2018) that looks at both FGSM and PGD (a stronger attack) on completely different networks and tasks (MNIST and CIFAR10). The paper would be stronger if content on adversarial robustness was removed entirely.
OTHER MINOR COMMENTS
* The title includes the word "starfire," but it never appears again in the paper. The paper proposes no specific technique, so there isn't anything to name.
* Use the \ begin{appendix} command before you create the appendices and the \ end{appendix} command when you are done. You can then use \section normally and each section so-created will appear with a letter rather than a number.
* Figure 4 is very hard to read.