Over-parameterization is ubiquitous nowadays in training neural networks to benefit both optimization in seeking global optima and generalization in reducing prediction error. However, compressive networks are desired in many real world applications and direct training of small networks may be trapped in local optima. In this paper, instead of pruning or distilling over-parameterized models to compressive ones, we propose a new approach based on differential inclusions of inverse scale spaces, that generates a family of models from simple to complex ones by coupling gradient descent and mirror descent to explore model structural sparsity. It has a simple discretization, called the Split Linearized Bregman Iteration (SplitLBI), whose global convergence analysis in deep learning is established that from any initializations, algorithmic iterations converge to a critical point of empirical risks. Experimental evidence shows that SplitLBI may achieve comparable and even better performance than other training algorithms on ResNet-18 in large scale training on ImageNet-2012 dataset etc., while with early stopping it unveils effective subnet architecture with comparable test accuracy to dense models after retraining instead of pruning well-trained ones.
Summary
=======
This paper aims to train sparse neural networks efficiently, by jointly optimizing the weights and sparsity structure of the network. It applies the Split Linear Bregman Iteration (Split LBI) method from [1] in a large-scale setting, to train deep neural networks.
The approach works by considering optimization in a joint space (W, \Gamma) consisting of the network weights W and a new set of parameters \Gamma that model the structural sparsity of the network.
The problem of learning sparse networks efficiently is important for modern applications that run on embedded devices, as well as for fast training on specialized hardware. I think the approach is interesting as a potential alternative to more expensive methods for finding sparse networks, such as NAS and the successive pruning & re-training approach of [2].
Overall, the paper pursues a promising direction to induce network sparsity, and presents some interesting results. However, there are several issues with the experiments and the structure/presentation of the paper that should be addressed.
Pros
====
* As far as I am aware, this is the first application of Split LBI to train deep neural networks.
* It shows that joint optimization of the weights and sparsity structure performs on par with baselines that only optimize weights, on MNIST, CIFAR-10, and ImageNet.
* It provides a global convergence analysis that shows that the weights optimized with Split LBI converge to a critical point of the training loss, regardless of initialization.
* It provides an ablation study for the two hyperparameters \kappa and \nu of Split LBI.
* I think the most interesting parts of the paper are those that examine the structural sparsity learned by Split LBI (Sections 4.3 and 4.4). In particular, the fact that Split LBI was able to match or outperform the test accuracy of several baselines (network slimming, soft filter pruning, and the method in Rethinking the Lottery Ticket Hypothesis) in a single training run (without re-training) is a nice result.
Issues
======
* The Split LBI method is presented as a novel contribution (in the abstract: "we propose a new approach based on differential inclusions of inverse scale spaces ..."), but it was already described in detail in a paper that they cite ([1]) published at NeurIPS 2016. The only theoretical contribution of this paper is section 3: Global convergence of Split LBI.
* The claim that "Split LBI demonstrates SOTA performance in large scale training on ImageNet" (from the abstract) is not correct, and needs to be qualified. This paper reports 70.55/89.56% (top 1 / top 5) accuracy; as far as I am aware, the current SOTA on ImageNet is [3], which achieves 86.4/98.0% (top 1 / top 5) accuracy. I think it would be better for this paper to argue that it achieves comparable performance to baselines with a particular architecture and training regime.
* The structure and presentation of the paper could be improved in several ways, outlined as follows:
    - Most of the Methodology section discusses the Split LBI method from prior work. I would encourage the authors to split the Methodology section into a separate Background section for Split LBI, followed by a new section specifically about applying Split-LBI to convolutional and fully-connected layers in neural networks.
    - The writing is missing some details and explanations that would be very helpful for readers. For example, it should clearly state that the dimension of \Gamma is the same as the dimension of the weights W.
    - It would also be good to expand the explanation about why SplitISS avoids the parameter correlation problem, and what it means for \Gamma to have an orthogonal design?
    - The figures are too small to be readable without a lot of zooming.
    - The paper ends abruptly, with no conclusion.
    - The appendix contains some useful material, but much of it is not referenced from the main paper. I think some parts of the appendix could be moved to the main paper, for example the comparison of computational and memory costs between Split LBI and SGD.
* I am not sure what the purpose of the comparisons between optimizers in Table 1 is. The motivation given in the abstract and introduction is to learn sparse networks online during optimization; it does not propose Split LBI as a new optimizer to compete with Adam. Couldn't one use the Adam update rule to optimize the weights in Split-LBI? I think it makes sense to compare Split LBI to standard training setups that do not enforce any sparsity, as well as to setups that use L1 and L2 regularization, but I do not think that Table 1 is set up correctly for this. Different optimizers are paired with different regularizers, and crucially the choice of hyperparameters is not discussed---how did you choose the coefficient of L1 regularization to be 1e-3? Additionally, many rows in Table 1 are missing data (e.g., the variants of Adam are only run for CIFAR-10).
* Regarding experiments, it is not clear which experiments the authors actually performed, for which they took the results from previously published papers, and sometimes where the results come from at all.
    - In Table 1, there is an asterix next to SGD-Mom-Wd that the authors say indicates "results from the official pytorch website." That would imply that the rest of the experiments were done by the authors (and the authors say in the caption "we use the official pytorch codes to run the competitors"). However, Table 2 (found in the Appendix, page 20), contains identical numbers and a sign # that, according to the authors, indicates results of their own experiments. That would mean that of all the SGD and Adam experiments shown in the table, the authors only performed SGD-naive and Adam-naive. Where do the other numbers come from? What does "official pytorch code to run the competitors" mean? Where is that code from?
    - Figure 4 contains baselines and SplitLBI results. Where do the numbers for the baselines come from? The caption mentions another paper, [5], but I did not find the source of the numbers in that paper. The caption seems to point to Table 9a in [5], but that table does not deal with Network Slimming, Soft-Filter Pruning, Scratch B, or Scratch-E. Additionally, Table 9a of [5] only contains results for VGG-16 and ResNet-50. Where do the baselines for ResNet-56 (Figure 4b) come from?
* How is the proximal objective in Eq. 5 optimized? That is, how do you compute the argmin?
* Figure 2 shows results for SLBI-1 and SLBI-10, but no discussion of what SLBI-1 and SLBI-10 mean. Also regarding Fig.2, the authors claim that "Filters learned by ImageNet prefer non-semantic texture rather than shape and color." How did the authors come to this conclusion? I looked carefully at the filter visualizations, and I cannot see a clear difference between the filters learned by Split LBI and SGD.
* The computation time comparison in Table 11 (Appendix E) is a bit strange, because it shows that Adam takes 2x as long as SGD, which does not align with my experience; in practice, the wall-clock time is nearly identical between Adam and SGD. It would be good to provide more details about how the time was measured. Also, does the memory comparison measure only the memory used for model parameters (W and \Gamma), or also activation memory? Shouldn't Split LBI use 2x the memory of SGD (if measuring only the weights)?
* In Figure 1, it looks like the initial magnitude of the filters is larger for SGD compared to Split LBI. Are the weights initialized in the same way? Also, why is the setup of the MNIST experiment in Fig.1 different from the setup in Table 1 (e.g., learning rate decay every 40 epochs vs every 30 epochs)? In addition, it looks like the first learning rate decay causes the filter weight magnitudes to flatten out and stay constant.
* What is the learning rate schedule used for the runs in Figure 3? It looks like the lr decays at epoch 80 and 120, but this is only mentioned in table captions in the appendix. This should be stated in the main paper. Also, why is this a different training setup from that used for Table 1?
I also noted that the authors do not intend to make the code public upon publication of the paper. On page 6, they state that "source codes will be released upon requests." At present, the preferred path is to make the code public upon publication of the paper.
Minor points
============
* In the caption of Figure 3, it says "The results are repeated for 5 times. Shaded area indicates the variance; and in each round, we keep the exactly same initialization for each model." What is different between the 5 runs if the initialization is the same?
* There are too many different colors used in Figures 1 and 2. Since the purple, green, and black boxes are important to see for figures 1 and 2, it is confusing to have to deal with additional blue, pink, and yellow boxes around every three.
[1] Huang et al., Split LBI: An iterative regularization path with structural sparsity. NeurIPS 2016.
[2] Frankle & Carbin, The Lottery Ticket Hypothesis: Finding sparse, trainable neural networks. ICLR 2019.
[3] Touvron et al., Fixing the train-test resolution discrepancy. https://arxiv.org/abs/1906.06423.
[4] He et al., Deep residual learning for image recognition. https://arxiv.org/abs/1512.03385.
[5] Liu et al., Rethinking the value of network pruning. ICLR 2019.
Post-rebuttal Update
====================
I thank the authors for their rebuttal, and for clarifying some details in the paper.
* I think the experiments on sparsity are interesting. More efficient ways to find good sparse networks are certainly of interest to the community.
* I appreciate that the authors released the source code.
* In summary, this paper applies Split-LBI to neural network training, and provides a global convergence result as one of the main contributions. Operationally, compared to the original Split LBI approach, it changes the loss function from squared error to cross entropy, and uses mini-batches for training, which are fairly straightforward.
* One important issue with the paper is that it blurs the distinction between prior work and the new contribution. For example, the subsection on Split Linearized Bregman Iteration in the "Methodology" section does not contain anything new compared to [1], and this is not clear enough to the reader. Also, not enough credit is given to [2] for the "Differential of Inclusion of Inverse Scale Space" subsection. I maintain that there needs to be a separate "Background" section for this, and that it should be made absolutely clear what is new in the "Methodology" section. It feels like this distinction is obfuscated in the writing.
* Given that the authors propose Split-LBI as a new optimizer that can be compared to others (e.g., Adam), one issue is that there doesn't seem to be any search done over the hyperparameters of each optimizer, including the learning rate and amount of weight decay. For example, Adam is only run with learning rate 1e-3 and SGD is run with 0.1; in addition, the weight decay (where used) is only set to 1e-4. Thus, it is not clear how meaningful these comparisons are. Also, in Table 1, the CIFAR-10 test accuracies are fairly low at ~90%, while modern models such as Wide ResNets can achieve ~95%.
* The newly-written conclusion is still incorrect, stating again that Split LBI achieves SOTA performance on ImageNet. Also, if  "with better interpretability than SGD" refers to the qualitative comparison of the learned filters, I think this conclusion is a bit too strong, because I don't think the difference is visible enough to aid interpretability.
* Minor point: On further inspection, the legend in the left-side plots in Figure 2 does not match the labels of the visualizations on the right. There is no yellow training curve in Figure 2, despite the assertion in the rebuttal.
[1] Huang et al., Split LBI: An iterative regularization path with structural sparsity. NeurIPS 2016.
[2] Osher et al., "Sparse recovery via differential inclusions." Applied and Computational Harmonic Analysis, 2016.
I maintain my score of weak reject, but am not totally opposed to it being accepted, because it provides a way to find sparse networks more efficiently.