In search of theoretically grounded pruning
Deep learning relies on resource-heavy linear algebra operations which can be prohibitively expensive when deploying to constrained embedded and mobile devices, or even when training large-scale networks. One way to reduce a neural network*s resource requirements is to sparsify its weight matrices - a process often referred to as pruning. It is typically achieved by removing least important weights as measured by some salience criterion, with pruning by magnitude being the most popular option. This, however, often makes close to random judgments. In this paper we aim to closely investigate the concept of model weight importance, with a particular focus on the magnitude criterion and its most suitable substitute. To this end we identify a suitable Statistical framework and derive deep model parameter asymptotic theory to use with it. Thus, we derive a statistically-grounded pruning criterion which we compare with the magnitude pruning both qualitatively and quantitatively. We find this criterion to better capture parameter salience, by accounting for its estimation uncertainty. This results in improved performance and easier post-pruned re-training.
 Comments * all conditions are not discussed, for instance, isn*t it quite strong to impose that the parameter space .Theta is compact? * LDH condition: what is .mathcal{N}, it isn*t defined * The statement on top of page 3: "Then, asymptotically, as the sample size n grows to infinity, the model parameters are distributed as (...)" is awkward: is it standard? is it a new result? please consider stating it as a Proposition/Theorem or something, and explain where it comes from. * Actually, I am skeptical about the above statement itself: it is stated that "the model parameters are distributed as (...)". Does that mean that the parameters are random, in a Bayesian fashion? please state it then. Or the alluded to distribution is that of parameters* estimators? then the statement should be amended, and it should be explained what estimators are considered. * footnote 3: shouldn*t the considered sum be divided by n? * the paper is full of flawed equations where, I think, parameters are mixed up with estimators. * For instance, Equations (2) and (3) should express the same quantity, , however the former displays the parameters’ covariance matrix, while the latter uses an estimator of it, .hat{Avar}. * Same in page 4 with the estimator of the Hessians (H) and of the score covariance matrix (.hat Σ). * Same for Equation (6): .hat .theta is suddenly plugged-in in place of .theta_0, all of this conserving the equality sign. * Same for Equation (7) Typos, minor comments and un-precise statements * the footnotes superscripts are misleading for footnotes 2 and 3; they could be better placed. * Lindeberg-Lvy should be Lindeberg-Lévy * after equations (6,7,8) you write "Where all is defined as above": this is fairly un-precise; as you use LaTeX, you should conveniently use some .label and .ref. * end of section 4.1: MINST dataset, a new dataset? * top of section 4.2: there are even typos in the proposed name for methodology! : "Walt-based Pruning" * Many references in the bibliography are incomplete, so that it is impossible to see where they come from (at least seven of them)
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
In search of theoretically grounded pruning
Deep learning relies on resource-heavy linear algebra operations which can be prohibitively expensive when deploying to constrained embedded and mobile devices, or even when training large-scale networks. One way to reduce a neural network*s resource requirements is to sparsify its weight matrices - a process often referred to as pruning. It is typically achieved by removing least important weights as measured by some salience criterion, with pruning by magnitude being the most popular option. This, however, often makes close to random judgments. In this paper we aim to closely investigate the concept of model weight importance, with a particular focus on the magnitude criterion and its most suitable substitute. To this end we identify a suitable Statistical framework and derive deep model parameter asymptotic theory to use with it. Thus, we derive a statistically-grounded pruning criterion which we compare with the magnitude pruning both qualitatively and quantitatively. We find this criterion to better capture parameter salience, by accounting for its estimation uncertainty. This results in improved performance and easier post-pruned re-training.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 Comments * all conditions are not discussed, for instance, isn*t it quite strong to impose that the parameter space .Theta is compact? * LDH condition: what is .mathcal{N}, it isn*t defined * The statement on top of page 3: "Then, asymptotically, as the sample size n grows to infinity, the model parameters are distributed as (...)" is awkward: is it standard? is it a new result? please consider stating it as a Proposition/Theorem or something, and explain where it comes from. * Actually, I am skeptical about the above statement itself: it is stated that "the model parameters are distributed as (...)". Does that mean that the parameters are random, in a Bayesian fashion? please state it then. Or the alluded to distribution is that of parameters* estimators? then the statement should be amended, and it should be explained what estimators are considered. * footnote 3: shouldn*t the considered sum be divided by n? * the paper is full of flawed equations where, I think, parameters are mixed up with estimators. * For instance, Equations (2) and (3) should express the same quantity, , however the former displays the parameters’ covariance matrix, while the latter uses an estimator of it, .hat{Avar}. * Same in page 4 with the estimator of the Hessians (H) and of the score covariance matrix (.hat Σ). * Same for Equation (6): .hat .theta is suddenly plugged-in in place of .theta_0, all of this conserving the equality sign. * Same for Equation (7) Typos, minor comments and un-precise statements * the footnotes superscripts are misleading for footnotes 2 and 3; they could be better placed. * Lindeberg-Lvy should be Lindeberg-Lévy * after equations (6,7,8) you write "Where all is defined as above": this is fairly un-precise; as you use LaTeX, you should conveniently use some .label and .ref. * end of section 4.1: MINST dataset, a new dataset? * top of section 4.2: there are even typos in the proposed name for methodology! : "Walt-based Pruning" * Many references in the bibliography are incomplete, so that it is impossible to see where they come from (at least seven of them)
assistant
 Comments * all conditions are not discussed, for instance, isn*t it quite strong to impose that the parameter space .Theta is compact? * LDH condition: what is .mathcal{N}, it isn*t defined * The statement on top of page 3: "Then, asymptotically, as the sample size n grows to infinity, the model parameters are distributed as (...)" is awkward: is it standard? is it a new result? please consider stating it as a Proposition/Theorem or something, and explain where it comes from. * Actually, I am skeptical about the above statement itself: it is stated that "the model parameters are distributed as (...)". Does that mean that the parameters are random, in a Bayesian fashion? please state it then. Or the alluded to distribution is that of parameters* estimators? then the statement should be amended, and it should be explained what estimators are considered. * footnote 3: shouldn*t the considered sum be divided by n? * the paper is full of flawed equations where, I think, parameters are mixed up with estimators. * For instance, Equations (2) and (3) should express the same quantity, , however the former displays the parameters’ covariance matrix, while the latter uses an estimator of it, .hat{Avar}. * Same in page 4 with the estimator of the Hessians (H) and of the score covariance matrix (.hat Σ). * Same for Equation (6): .hat .theta is suddenly plugged-in in place of .theta_0, all of this conserving the equality sign. * Same for Equation (7) Typos, minor comments and un-precise statements * the footnotes superscripts are misleading for footnotes 2 and 3; they could be better placed. * Lindeberg-Lvy should be Lindeberg-Lévy * after equations (6,7,8) you write "Where all is defined as above": this is fairly un-precise; as you use LaTeX, you should conveniently use some .label and .ref. * end of section 4.1: MINST dataset, a new dataset? * top of section 4.2: there are even typos in the proposed name for methodology! : "Walt-based Pruning" * Many references in the bibliography are incomplete, so that it is impossible to see where they come from (at least seven of them)