Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization
Modern deep neural networks are highly overparameterized, and often of huge sizes. A number of post-training model compression techniques, such as distillation, pruning and quantization, can reduce the size of network parameters by a substantial fraction with little loss in performance. However, training a small network of the post-compression size de novo typically fails to reach the same level of accuracy achieved by compression of a large network, leading to a widely-held belief that gross overparameterization is essential to effective learning. In this work, we argue that this is not necessarily true. We describe a dynamic sparse reparameterization technique that closed the performance gap between a model compressed through iterative pruning and a model of the post-compression size trained de novo. We applied our method to training deep residual networks and showed that it outperformed existing reparameterization techniques, yielding the best accuracy for a given parameter budget for training. Compared to existing dynamic reparameterization methods that reallocate non-zero parameters during training, our approach achieved better performance at lower computational cost. Our method is not only of practical value for training under stringent memory constraints, but also potentially informative to theoretical understanding of generalization properties of overparameterized deep neural networks.
 The paper provides a dynamic sparse reparameterization method allowing small networks to be trained at a comparable accuracy as pruned network with (initially) large parameter spaces. Improper initialization along with a fewer number of parameters requires a large parameter model, to begin with (Frankle and Carbin, 2018). The proposed method which is basically a global pooling followed by a tensorwise growth allocates free parameters using an efficient weight re-distribution scheme, uses an approximate thresholding method and provides automatic parameter re-allocation to achieve its goals efficiently. The authors empirically demonstrate their results on MNIST, CIFAR-10, and Imagenet and show that dynamic sparse provides higher accuracy than compressed sparse (and other) networks. The paper is addressing an important problem where instead of training and pruning, directly training smaller networks is considered. In that respect, the paper does provide some useful tricks to reparameterize and pick the top filters to prune. I especially enjoyed reading the discussion section. However, the hyperbole in claims such as "first dynamic reparameterization method for training convolutional network" makes it hard to judge the paper favorably given previous methods that have already proposed dynamic reparameterization and explored. This language is consistent throughout the paper and the paper needs a revision that positions this paper appropriately before it is accepted. The proposed technique provides limited but useful contributions over existing work as in SET and DeepR. However, an empirical comparison against them in your evaluation section can make the paper stronger especially if you claim your methods are superior. How does your training times compare with the other methods? Re-training times are a big drawback of pruning methods and showing those numbers will be useful.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization
Modern deep neural networks are highly overparameterized, and often of huge sizes. A number of post-training model compression techniques, such as distillation, pruning and quantization, can reduce the size of network parameters by a substantial fraction with little loss in performance. However, training a small network of the post-compression size de novo typically fails to reach the same level of accuracy achieved by compression of a large network, leading to a widely-held belief that gross overparameterization is essential to effective learning. In this work, we argue that this is not necessarily true. We describe a dynamic sparse reparameterization technique that closed the performance gap between a model compressed through iterative pruning and a model of the post-compression size trained de novo. We applied our method to training deep residual networks and showed that it outperformed existing reparameterization techniques, yielding the best accuracy for a given parameter budget for training. Compared to existing dynamic reparameterization methods that reallocate non-zero parameters during training, our approach achieved better performance at lower computational cost. Our method is not only of practical value for training under stringent memory constraints, but also potentially informative to theoretical understanding of generalization properties of overparameterized deep neural networks.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 The paper provides a dynamic sparse reparameterization method allowing small networks to be trained at a comparable accuracy as pruned network with (initially) large parameter spaces. Improper initialization along with a fewer number of parameters requires a large parameter model, to begin with (Frankle and Carbin, 2018). The proposed method which is basically a global pooling followed by a tensorwise growth allocates free parameters using an efficient weight re-distribution scheme, uses an approximate thresholding method and provides automatic parameter re-allocation to achieve its goals efficiently. The authors empirically demonstrate their results on MNIST, CIFAR-10, and Imagenet and show that dynamic sparse provides higher accuracy than compressed sparse (and other) networks. The paper is addressing an important problem where instead of training and pruning, directly training smaller networks is considered. In that respect, the paper does provide some useful tricks to reparameterize and pick the top filters to prune. I especially enjoyed reading the discussion section. However, the hyperbole in claims such as "first dynamic reparameterization method for training convolutional network" makes it hard to judge the paper favorably given previous methods that have already proposed dynamic reparameterization and explored. This language is consistent throughout the paper and the paper needs a revision that positions this paper appropriately before it is accepted. The proposed technique provides limited but useful contributions over existing work as in SET and DeepR. However, an empirical comparison against them in your evaluation section can make the paper stronger especially if you claim your methods are superior. How does your training times compare with the other methods? Re-training times are a big drawback of pruning methods and showing those numbers will be useful.
assistant
 The paper provides a dynamic sparse reparameterization method allowing small networks to be trained at a comparable accuracy as pruned network with (initially) large parameter spaces. Improper initialization along with a fewer number of parameters requires a large parameter model, to begin with (Frankle and Carbin, 2018). The proposed method which is basically a global pooling followed by a tensorwise growth allocates free parameters using an efficient weight re-distribution scheme, uses an approximate thresholding method and provides automatic parameter re-allocation to achieve its goals efficiently. The authors empirically demonstrate their results on MNIST, CIFAR-10, and Imagenet and show that dynamic sparse provides higher accuracy than compressed sparse (and other) networks. The paper is addressing an important problem where instead of training and pruning, directly training smaller networks is considered. In that respect, the paper does provide some useful tricks to reparameterize and pick the top filters to prune. I especially enjoyed reading the discussion section. However, the hyperbole in claims such as "first dynamic reparameterization method for training convolutional network" makes it hard to judge the paper favorably given previous methods that have already proposed dynamic reparameterization and explored. This language is consistent throughout the paper and the paper needs a revision that positions this paper appropriately before it is accepted. The proposed technique provides limited but useful contributions over existing work as in SET and DeepR. However, an empirical comparison against them in your evaluation section can make the paper stronger especially if you claim your methods are superior. How does your training times compare with the other methods? Re-training times are a big drawback of pruning methods and showing those numbers will be useful.