The Lottery Ticket Hypothesis from  conjectures that, for typically-sized neural networks, it is possible to find small sub-networks which train faster and yield superior performance than their original counterparts. The proposed algorithm to search for "winning tickets", Iterative Magnitude Pruning, consistently finds sub-networks with 90 − 95% less parameters which train faster and better than the overparameterized models they were extracted from, creating potential applications to problems such as transfer learning. In this paper, we propose Continuous Sparsification, a new algorithm to search for winning tickets which continuously removes parameters from a network during training, and learns the sub-network's structure with gradient-based methods instead of relying on pruning strategies. We show empirically that our method is capable of finding tickets that outperforms the ones learned by Iterative Magnitude Pruning, and at the same time providing faster search, when measured in number of training epochs or wall-clock time.
To the authors of paper 2504: I have posted a private comment for the reviewer/AC discussion period based on your author responses and your revised paper. I want you to be able to see my full response, but I can't post additional public comments to the paper. As such, I'm editing my review with exactly what I sent to the AC.
Thank you to the authors for your thoughtful rebuttal and for updating the paper with both new text and (more impressively) new experiments. I read all three of your rebuttals and re-read the revised paper in detail. My comment to the AC is at the bottom of this message. I don't know if you will get an alert that the review was updated, but I hope you get a chance to take a look.
===========================================
EXECUTIVE SUMMARY OF REVIEW
Summary of paper: This paper proposes a technique that simultaneously trains a neural network and learns a pruning mask. The goal of this technique is to make it faster to retroactively find sparse subnetworks that, from a point early in training, could have trained in isolation to the same performance as the full network ("winning tickets"). The best subnetworks found by this technique outperform those found by existing techniques [1] at every sparsity level.
Summary of review: The technique introduces several new hyperparameters whose values are asserted without describing the extent of the search necessary to find them; it unclear whether the cost of the search cancels out the efficiency gains. In addition, the proposed technique is inconsistent across runs. The sparsity and accuracy of the subnetworks it produces vary greatly, and there are no hyperparameters to explicitly control these outcomes. It is also unclear whether results vary from run to run even with the same hyperparameters.  As such, this technique is not clearly a cost reduction as compared to existing approaches in [1] when it comes to studying lottery tickets. Moreover, the paper implicitly proposes a new pruning technique, and it should be evaluated against other related techniques in the pruning literature. Finally, the evaluation needs more rigor.
Conclusion: It is difficult to determine whether the technique is an improvement over existing methods since the true costs of using it in practical workflows are unclear. Weak reject.
Opportunity to improve score: Include a more detailed analysis of the overall costs of finding winning tickets at a target sparsity using the proposed technique, particularly including the costs of hyperparameter search necessary to do so. Include multiple replicates of experiments, experiments on more networks, and information about the variance of performance across runs with the same hyperparameters. Compare against other state-of-the-art pruning techniques.
PROBLEM STATEMENT AND PROPOSED SOLUTION
Problem: Winning tickets are currently expensive to find. The best known procedure [1, 2] is "iterative magnitude pruning" (IMP), which involves repeatedly training a network to completion, pruning by a fixed percentage, and "rewinding" weights to an early iteration of training until the network reaches the desired level of sparsity. To reach sufficient sparsity on standard networks, this procedure must be repeated 10 or more times.
Goal: To propose a procedure that finds winning tickets more efficiently.
Significance: A more efficient procedure would make it easier to study the lottery ticket phenomenon. (Whether studying that phenomenon is, itself, significant is debatable.) Personally, I have extensive experience using IMP to find winning tickets, so techniques to reduce the cost of finding winning tickets would be very valuable for my work.
Proposed solution: The authors propose "continuous sparsification" (CS), which makes it possible to learn which weights to prune simultaneously with the weights themselves. Accompanying each parameter w in the network is a second parameter s. The actual weight used in the network is w * sigmoid(s * beta), where beta is a temperature hyperparameter. If the learned value of s is such that sigmoid(s * beta) is approximately 0 at the end of training, then the parameter has been pruned. The value of beta increases exponentially throughout training, meaning the output of the sigmoid will be closer to a hard 0 or 1, producing a pruning mask. To ensure sparsification happens, a regularization term lambda |sigmoid(beta * s)| is added. When run over multiple iterations, values of s are reset to their original values for weights that are not pruned. In addition, weights are either rewound (as in IMP) or left at their final values (as in [3]).
Novelty: The paper is slightly novel. The technique is a variation of those proposed in [4] and [5], but the changes are meaningful. This is the first known use for finding winning tickets, but any pruning technique could hypothetically be used for this purpose. In effect, the paper proposes a new pruning technique that is primarily evaluated for its efficacy in finding winning tickets.
TECHNICAL REVIEW
* This technique introduces several new hyperparameters: lambda, initial and final betas, and the initial values for s. The paper suggests good values for these hyperparameters for both networks considered. These values were presumably found through hyperparameter search of some kind. How extensive was this hyperparameter search, and what is the range of "good" combinations of values? This is not just a methodological footnote; this paper's stated goal is to improve the efficiency of finding winning lottery tickets, and - if good hyperparameters are hard to find - then that defeats the purpose of a more efficient technique. IMP, while far less efficient epoch for epoch as compared to CS, requires no hyperparameter search; global pruning 20% of parameters per iteration seems to work well in general [2]. In a revised version of the paper, I would be eager to learn more about this set of tradeoffs, since that is what matters in practice.
* The authors only study two networks: a toy convolutional network and a small Resnet. It is hard to draw broad conclusions from such a limited set of examples. I would be particularly interested in seeing how this technique performs on a large-scale network for ImageNet (e.g., Resnet-50), since these are the situations where IMP becomes particularly cost-prohibitive. If the technique works well in these settings, it would enable lottery ticket research at much larger scales than is currently possible. If the technique works as efficiently at this scale, then doing could even be feasible during the rebuttal period. (I acknowledge getting experiments working on ImageNet is no small undertaking in terms of both engineering time and cost, but it would improve my confidence to see those results.)
* The authors did an admirably careful job replicating the networks in [1], which include a variety of nonstandard hyperparameters.
* Did the paper study Resnet-18 (a network designed for ImageNet with 11.2M parameters) or Resnet-20 (a network designed for CIFAR-10 with 272K parameters)? Frankle et al. [1, 2] describe Resnet-20 in their appendices but mistakenly refer to it as Resnet-18 throughout both papers, so I wanted to clarify. Based on the final test accuracy of the network, it appears to be Resnet-20; if so, I'd urge you to call it as such and note in a parenthetical or footnote that it's the same network as in [1, 2] but with Frankle et al's mistaken name corrected.
* Are the values of s0 for CS sampled from a distribution, or are they all the same, fixed value?
* Do the extra parameters lead to longer wall-clock training times? If you are making an argument about a more efficient technique, this is an important consideration.
* Figure 2 includes the same graph twice. I believe a different graph should appear on the left, and I am eager to take a look at it in a revised version of the paper.
* It does not appear that multiple replicates of each experiment were run with different network initializations in Figure 3. There don't appear to be any error bars on Figure 3, suggesting that this only represents a single initialization. Considering the wide variance in performance achieved by continuous sparsification across runs in Figure 3 (right), this graph needs to include multiple runs and error bars. (I assume the multiple runs shown in Figure 3 (right) are with different hyperparameters?)
* What is the performance of CS as a pruning technique? By finding winning tickets, CS is also implicitly pruning the network. Is this competitive with L0 Regularization [4]? Is it more efficient than iterative magnitude pruning as in [3]? If this is a more efficient technique for finding winning tickets, it is also liable to be a more efficient pruning method, which seems like even broader impact for the proposed technique. Alternatively, if this technique is less effective than comparable work (especially [4]) as a pruning method, then it is possible comparable work (especially [4]) might also produce better winning tickets, in which case the importance of this work is diminished. In an updated version of the paper, I would be interested in seeing an evaluation of CS as a pruning technique independent of the lottery ticket hypothesis (and compared to standard techniques in the pruning literature as such). I have a hard time seeing any reason why new techniques for finding lottery tickets are any different than new pruning techniques, and they should be evaluated as such in the context of the broader literature on pruning.
* The last paragraph of Section 4.1 is missing important details that are necessary to evaluate the utility of continuous sparsification in comparison to IMP. "In only two iterations, CS finds a ticket with over 77% sparsity..." - for which hyperparameters, and how many hyperparameters had to be explored to find these values? How was the pareto curve obtained? How many different runs were necessary to create it? How many total epochs of training did it take to find that curve? If obtaining this pareto curve required many runs of CS, then it may not be any more efficient than running IMP in practice. The pareto curves appear to make CS look misleadingly effective, since they hide many of the actual costs involved (e.g., hyperparameter search, number of separate runs that were conducted to produce the curve, etc.). Greater transparency of this aspect of the paper would go a long way toward increasing my confidence that the findings are an improvement over IMP. 
* The sparsity and accuracy of subnetworks found by CS appears to vary widely from run to run as shown in Figure 3 (right). The final sparsity appears to be a function of the values of s0, lambda, and beta, potentially along with luck from the optimization process. For the same values of s, lambda, and beta, how widely do the final sparsities and accuracies vary? If I wanted to find a winning ticket with a particular sparsity using CS, what would the procedure look like for doing so? Would I have to sweep across values of these hyperparameters, or is there a more straightforward way to do so? The practical usefulness of the procedure hinges on these questions.
* "We associate the performance drop of highly sparse tickets found by our method from the second iteration onwards to the lack of weight rewinding." Why didn't you also try it with weight rewinding? That seems like an easy way to evaluate this hypothesis. For both networks, it would be interesting to see the performance of CS with and without rewinding (analogous to Appendix B in [1]).
* In the literature so far, the only winning tickets to be examined are those from IMP [1, 2]. CS is a different technique, and it likely finds different winning tickets. Do these winning tickets have different properties from [1, 2]? Can they be reinitialized? Can they be rewound earlier? These comparisons seem like an interesting scientific opportunity.
* There are a few additional comparisons that I think are vital to include in the paper to appropriately contextualize results.  They're in the bullets below.
* First comparison: it would be useful to include random reinitialization or random pruning baselines as in [1, 2] simply to make it easier for the reader to contextualize the performance of other sparse subnetworks.
* Second comparison: what happens if you run IMP such that each iteration prunes to the same sparsity as achieved by each iteration of CS? Perhaps pruning by a fixed amount per iteration in IMP is wasteful, and one can prune more aggressively during earlier iterations as CS naturally appears to do. In other words, one way of explaining the advantage of CS would be that it prunes more aggressively. Is this indeed the case? I would be very curious to know.
* Third comparison: What are the range of results achieved if IMP and CS are run "one-shot" (pruning after just one iteration; in the case of IMP, pruning directly to a desired sparsity)? That is, how well can these techniques do with just a single iteration?
WRITING
The writing is excellent. The prose is clear, and I was able to fully understand a relatively sophisticated technique on the first read through the paper. Writing of this quality is rare, and the authors should be commended for it.
OVERALL: Weak Reject
The problem statement is that IMP is not efficient. The paper claims that CS is more efficient. However, the paper does not present a convincing case that CS is, on the whole, more efficient when taking into account hyperparameter search to get CS to work, hyperparameter search to target a particular sparsity, potential variance across runs of CS, potential additional training costs of CS, and the possibility that IMP might be able to work comparable well given a more aggressive pruning schedule.
In addition, the evaluation needs more experiments, including multiple replicates for each experiment and more networks (ideally one on ImageNet).
Finally, I am unclear on what distinguishes CS from any other pruning technique, and it should be evaluated in the context of the broader pruning literature.
If the authors clarify that the overall cost of CS (including all of the factors listed above) is lower than for IMP, address technical concerns about the evaluation, and evaluate CS as a pruning technique in an updated version of the paper, I will update my score accordingly.
[1] Frankle & Carbin. "The Lottery Ticket Hypothesis." ICLR 2019.
[2] Frankle et al. "Stabilizing the Lottery Ticket Hypothesis." Arxiv.
[3] Han et al. "Learning both Weights and Connections for Efficient Neural Networks." NeurIPS 2015.
[4] Louizos et al. "Learning Sparse Neural Networks through L0 Regularization." ICLR 2018
[5] Zhou et al. "Deconstructing Lottery Tickets: Signs, Zeros, and the SuperMask." NeurIPS 2019.
=====================================
COMMENT TO THE AC AFTER READING REBUTTALS AND REVISED PAPER
TLDR
I believe the technique is novel and should be published, regardless of whether it is actually more efficient than IMP for practical workflows.
However, I believe the current evaluation is inadequate, both in the experiments and in the way data is presented. Namely, it is impossible to actually compare the costs of CS and IMP in the scenarios the authors evaluate, despite the fact that efficiency is the authors' claimed contribution. From the data as presented, it is unclear whether CS is actually more efficient than IMP for scientific use-cases. I'm not particularly concerned if there isn't an efficiency advantage - it's an innovative contribution regardless. My concern is that it is impossible to compare the costs of these techniques given the current presentation of data. That speaks to flaws in the evaluation section.
I therefore maintain my score: weak reject. The technique deserves to be published, but the paper in its current form does not.
INTRODUCTION
After reading the original submission, I had the following questions:
1) Right now, the only way to find winning lottery tickets is through training the network and pruning it. IMP can be instantiated with any pruning technique, and the authors are really proposing a new pruning technique for use in the IMP framework. How does continuous sparsification perform as a pruning technique for network compression (i.e., independent of the lottery ticket hypothesis)?
2) How efficient is continuous sparsification (CS) in the scientific use cases where one would seek to find winning lottery tickets? In my research experience, there are two such use cases: (a) producing a winning lottery ticket with a specific sparsity and (b) producing winning lottery tickets across the full range of different sparsities.
In their reubttal, the authors addressed these underlying questions:
AS A PRUNING TECHNIQUE
"We have added new experiments showing that our method yields competitive results when pruning VGG trained on CIFAR, outperforming both magnitude pruning and stochastic l0 regularization"
My response: The initial results that the authors present are quite impressive on a VGG-style network for CIFAR-10. However, I find it concerning that the authors study pruning on this specific model but no others, particularly because they focus on a different network (namely, Resnet-20) in all other experiments in the paper. Since the authors are, in essence, proposing a new pruning technique, I would like to see it comprehensively evaluated as such on a range of networks against other baselines (as the authors do in Figure 4 for one network - those baselines are fine to me).
EFFICIENCY OF CONTINUOUS SPARSIFICATION
- It increases the cost of each individual network training run slightly: "Continuous Sparsification on a 1080 Ti: our method resulted in 15% extra wall-clock time per training epoch."
- PRODUCING WINNING TICKETS ACROSS THE FULL RANGE OF SPARSITIES: Throughout the paper, the authors present "pareto curves" showing the highest accuracy achieved by CS subnetworks at various sparsities. I find these pareto curves misleading: they hide the fact that CS had to be run several times with different hyperparameters to produce these curves.
For example, in Figure 6 (right), the authors produce the pareto curves by training Resnet-20 (by my count) 22 times (each point along the purple lines). The corresponding IMP curve appears to have 14 points. In other words, to get winning lottery tickets across all sparsities, CS must be run for more iterations than IMP - it appears to be less efficient, contradicting the authors' core claim. The best argument in favor of CS is that one could perform each of these runs in parallel if sufficient GPUs were available, meaning less wall-clock time would be required.
One caveat to this analysis: the authors state that they run CS "without rewinding," meaning that the second iteration of CS requires less training time than fully training the network (as IMP requires). The authors do not state how long they train when they *aren't* rewinding, so it is impossible to compare the efficiency of CS with IMP. All they say is that it "allow[s] for even faster ticket search." They also do not study IMP without rewinding, which would be a helpful baseline for comparing to CS without rewinding.
***In short, if the authors are making an argument that one technique is more efficient than the other on an epoch-for-epoch basis, they need to actually plot the epochs required by each technique.***
- PRODUCING A WINNING TICKET AT A SPECIFIC SPARSITY: My concern about this use case is that there is a non-intuitive relationship between the initialization of the sparsity parameters and the final sparsity of the network. As the authors state in the rebuttal: "To achieve a desired sparsity, one can either perform runs in parallel with different values for s_0, or perform sequential binary search if the goal is to minimize the overall computational cost and not wall-clock time." In other words, there is no precise way to target a particular sparsity other than trying many hyperparameter configurations. The authors do not provide any concrete costs of using CS in this way in comparison to IMP, and - from a usability perspective - this is a challenging workflow.
OTHER CONCERNS
- The authors only study CS on one toy network (the six-layer convolutional network, which - in my experience - is a particularly easy setting compared to deeper networks) and one "real" network (Resnet-20 on CIFAR-10). I would like to see results on other networks for CIFAR-10 (for example, the VGG network in Section 4.3).
- More importantly, I would like to see results on an ImageNet network. If CS makes finding winning lottery tickets more efficient as the authors claim, then finding winning tickets efficiently on an ImageNet network should be an excellent demonstration of their contribution. This scenario has stretched IMP to its breaking point, as the authors note.
ARGUMENTS IN FAVOR OF ACCEPTING
+ The authors propose a new pruning technique that appears to improve upon the increasingly popular L0-regularization technique.
+ With the right hyperparameters, the proposed technique makes it possible to find winning lottery tickets more efficiently than existing methods (i.e., IMP with magnitude pruning).
+ The winning tickets found by the proposed technique reach higher accuracy than IMP winning tickets and produce winning tickets at more extreme sparsities, improving upon our knowledge of the existence of winning lottery tickets.
ARGUMENTS IN FAVOR OF REJECTING
- The authors perform only minimal evaluation of their method as a pruning technique. It is possible that this is a missed opportunity to show an additional contribution. It is also possible that other, existing pruning techniques outperform CS at both pruning and finding winning lottery tickets.
- The technique is hard to use for the two existing use cases for finding winning tickets. In particular, there is no way to search for winning tickets at a specific sparsity.
- It is unclear whether CS is actually more efficient than IMP on an epoch-for-epoch basis, even though this is the main claimed contribution. The authors do not disclose - let alone plot - the number of training epochs required to find (a) winning tickets at a specific sparsity and (b) winning tickets at a range of sparsities, so it is impossible to make these comparisons. Meanwhile, the pareto curves the authors present are misleading, since they are the amalgamation of many separate runs of CS.
- The authors study their technique on only one "real" network (Resnet-20) for finding winning tickets and a separate "real" network (VGG-19) for pruning. The authors do not show how CS performs in other challenging settings, especially ImageNet.
- This work is only valuable if we believe that "lottery ticket hypothesis" work is valuable. In other words, this is a narrow contribution to an already-narrow area of study. This is one reason why pitching CS as a pruning technique would make this a stronger paper. I personally believe that "lottery ticket" work is valuable area of study, but I understand that it may not be seen as such in the broader ICLR community.
CONCLUSION
I believe the technique deserves to be published regardless of whether it is actually more efficient than IMP. It is a novel contribution to both our knowledge of pruning and of finding winning lottery tickets. The paper is exceptionally well written, and - as such - I believe it will help to inspire other research in this area.
However, I do not believe that the current paper - namely, the current evaluation - should be published. The paper presents no concrete data on the comparative costs of performing CS and IMP even though the core claim is that CS is more efficient. The paper does not disclose enough detail to compute these costs, and it seems like CS is more expensive than IMP for standard workflows. Moreover, the current presentation of the data through "pareto curves" is misleadingly favorable to CS.
I also believe that the paper needs experiments on ImageNet and needs a more thorough evaluation as a pruning technique beyond the lottery ticket hypothesis.
I therefore retain my current score of "weak reject," though I am eager to hear the thoughts of other reviewers, and I am open to changing my score.