Picking Winning Tickets Before Training by Preserving Gradient Flow
Overparameterization has been shown to benefit both the optimization and generalization of neural networks, but large networks are resource hungry at both training and test time. Network pruning can reduce test-time resource requirements, but is typically applied to trained networks and therefore cannot avoid the expensive training process. We aim to prune networks at initialization, thereby saving resources at training time as well. Specifically, we argue that efficient training requires preserving the gradient flow through the network. This leads to a simple but effective pruning criterion we term Gradient Signal Preservation (GraSP). We empirically investigate the effectiveness of the proposed method with extensive experiments on CIFAR-10, CIFAR-100, Tiny-ImageNet and ImageNet, using VGGNet and ResNet architectures. Our method can prune 80% of the weights of a VGG-16 network on ImageNet at initialization, with only a 1.6% drop in top-1 accuracy. Moreover, our method achieves significantly better performance than the baseline at extreme sparsity levels.
This paper introduces a method to prune networks at initialization in a way that (mostly) preserves the gradient flow through the resulting pruned network. This is a direct improvement over previous methods (e.g. SNIP) which have no guarantees that pruned connections will break the gradient flow and thereby harm learning. I quite like this paper, the motivation and results are convincing and it is well presented. The writing is excellent for most of the paper. From section 5 onwards the writing does need quite a bit of editing, as its quality is significantly reduced from what came before. Some detailed comments: - Figure 1 is very nice and really clarifies the idea! - In paragraph below Equation (8): what does *can be computed by backward twice* mean? - Please specify where the equalities in equation (9) are coming from. - Table 3 & 4: Why are the pruning ratios different for each model? - Table 3: Why are values missing for the baseline for 80% and 90%? - Section 5.2: *We observed that, the main bottleneck or pruned... when deriving the pruning criteria*: it*s not clear where this conclusion is coming from. - Table 5 has no batch size results, even though you*re referencing them in the text. And some minor comments to help with the writing: - Intro: *As shown in Dey et al. (2019) that with pre-specified sparsity, they can achieve* would read better as *As shown by Dey et al. (2019), with pre-specified sparsity one can achieve* - Equation (3): Clarify that this is a function of - Sentence below Equation (6): *of the pruned network, and thus our goal* remove the *and thus* - Table 1: Specify that you*re reporting accuracy. - Section 4.1: *e.g. wide ResNet (Zagaruyko & Komodakis, 2016), and thus we can regard* remove the *and thus* - Sentence below equation (9): *encouraging the eigenspace of Theta align* add a *to* before *align* - Sentence before section 5: *it will encourage the eigenspace of the NTK distributing large eigenvalues in the direction of Y, which will in turn accelerates the decrease of the loss (Arora et al., 2019) and benefits to the optimization in A* would read better as *it will encourage the eigenspace of the NTK to distribute large eigenvalues in the direction of Y, which in turn accelerates the decrease of the loss (Arora et al., 2019) and benefits the optimization in A* - Throughout section 5, write it in present tense rather than past tense. e.g. *In this section, we conduct various experiments* instead of *In this section, we conducted various experiments* - Sentence below table 2: you have *the the* - Second paragraph of section 5.1: *We can observe GraSP outperform random pruning clearly* would read better as *We can observe GraSP clearly outperforms random pruning* - Second paragraph of section 5.1: *In the next, we further compared* remove *In the next* - Second paragraph of section 5.1: *Besides, we further experimented with the late resetting* remove *Besides* - Paragraph above section 5.2: *GraSP surpassing SNIP* use *surpasses* instead - Paragraph above section 5.2: *investigate the reasons behind in Section 5.2 for promoting better understanding* would read better as *investigate the reasons behind this in Section 5.2 for obtaining a better understanding* - Section 5.2: *We observed that, the main bottleneck* -> *We observe that the main bottleneck* - Section 5.2: *Besides, we also plotted the the gradient norm of the pruned*, remove *Besides* and the extra *the* - Section 5.2: *the average of the gradients of the entire dataset* use *over the entire dataset* - Section 5.2: *hopefully more training progress can make as evidenced* would read better as *hopefully more training progress can be made as evidenced* - Section 5.3 title would be better using *Visualizing* instead of *Visualize* - Section 5.3: Join the first two sentences with a comma into a single sentence. - Section 5.3: *In contrast, SNIP are more likely* -> In contrast, SNIP is more likely* - Section 5.4: *for ablation study* would read better as *via ablations* - Section 5.4: *we tested GraSP with three different initialization methods;* use a *:* instead of *;* - Section 6: *Besides, readers may notice that*, remove the *Besides* - Section 6: *traditional pruning algorithms while still enjoy the cheaper training cost. As an evidence,* would read better as *traditional pruning algorithms while still enjoying cheaper training costs. As evidence,* - Your citation for Evci et al. (2019) is missing the publication venue/arxiv ID.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Picking Winning Tickets Before Training by Preserving Gradient Flow
Overparameterization has been shown to benefit both the optimization and generalization of neural networks, but large networks are resource hungry at both training and test time. Network pruning can reduce test-time resource requirements, but is typically applied to trained networks and therefore cannot avoid the expensive training process. We aim to prune networks at initialization, thereby saving resources at training time as well. Specifically, we argue that efficient training requires preserving the gradient flow through the network. This leads to a simple but effective pruning criterion we term Gradient Signal Preservation (GraSP). We empirically investigate the effectiveness of the proposed method with extensive experiments on CIFAR-10, CIFAR-100, Tiny-ImageNet and ImageNet, using VGGNet and ResNet architectures. Our method can prune 80% of the weights of a VGG-16 network on ImageNet at initialization, with only a 1.6% drop in top-1 accuracy. Moreover, our method achieves significantly better performance than the baseline at extreme sparsity levels.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This paper introduces a method to prune networks at initialization in a way that (mostly) preserves the gradient flow through the resulting pruned network. This is a direct improvement over previous methods (e.g. SNIP) which have no guarantees that pruned connections will break the gradient flow and thereby harm learning. I quite like this paper, the motivation and results are convincing and it is well presented. The writing is excellent for most of the paper. From section 5 onwards the writing does need quite a bit of editing, as its quality is significantly reduced from what came before. Some detailed comments: - Figure 1 is very nice and really clarifies the idea! - In paragraph below Equation (8): what does *can be computed by backward twice* mean? - Please specify where the equalities in equation (9) are coming from. - Table 3 & 4: Why are the pruning ratios different for each model? - Table 3: Why are values missing for the baseline for 80% and 90%? - Section 5.2: *We observed that, the main bottleneck or pruned... when deriving the pruning criteria*: it*s not clear where this conclusion is coming from. - Table 5 has no batch size results, even though you*re referencing them in the text. And some minor comments to help with the writing: - Intro: *As shown in Dey et al. (2019) that with pre-specified sparsity, they can achieve* would read better as *As shown by Dey et al. (2019), with pre-specified sparsity one can achieve* - Equation (3): Clarify that this is a function of - Sentence below Equation (6): *of the pruned network, and thus our goal* remove the *and thus* - Table 1: Specify that you*re reporting accuracy. - Section 4.1: *e.g. wide ResNet (Zagaruyko & Komodakis, 2016), and thus we can regard* remove the *and thus* - Sentence below equation (9): *encouraging the eigenspace of Theta align* add a *to* before *align* - Sentence before section 5: *it will encourage the eigenspace of the NTK distributing large eigenvalues in the direction of Y, which will in turn accelerates the decrease of the loss (Arora et al., 2019) and benefits to the optimization in A* would read better as *it will encourage the eigenspace of the NTK to distribute large eigenvalues in the direction of Y, which in turn accelerates the decrease of the loss (Arora et al., 2019) and benefits the optimization in A* - Throughout section 5, write it in present tense rather than past tense. e.g. *In this section, we conduct various experiments* instead of *In this section, we conducted various experiments* - Sentence below table 2: you have *the the* - Second paragraph of section 5.1: *We can observe GraSP outperform random pruning clearly* would read better as *We can observe GraSP clearly outperforms random pruning* - Second paragraph of section 5.1: *In the next, we further compared* remove *In the next* - Second paragraph of section 5.1: *Besides, we further experimented with the late resetting* remove *Besides* - Paragraph above section 5.2: *GraSP surpassing SNIP* use *surpasses* instead - Paragraph above section 5.2: *investigate the reasons behind in Section 5.2 for promoting better understanding* would read better as *investigate the reasons behind this in Section 5.2 for obtaining a better understanding* - Section 5.2: *We observed that, the main bottleneck* -> *We observe that the main bottleneck* - Section 5.2: *Besides, we also plotted the the gradient norm of the pruned*, remove *Besides* and the extra *the* - Section 5.2: *the average of the gradients of the entire dataset* use *over the entire dataset* - Section 5.2: *hopefully more training progress can make as evidenced* would read better as *hopefully more training progress can be made as evidenced* - Section 5.3 title would be better using *Visualizing* instead of *Visualize* - Section 5.3: Join the first two sentences with a comma into a single sentence. - Section 5.3: *In contrast, SNIP are more likely* -> In contrast, SNIP is more likely* - Section 5.4: *for ablation study* would read better as *via ablations* - Section 5.4: *we tested GraSP with three different initialization methods;* use a *:* instead of *;* - Section 6: *Besides, readers may notice that*, remove the *Besides* - Section 6: *traditional pruning algorithms while still enjoy the cheaper training cost. As an evidence,* would read better as *traditional pruning algorithms while still enjoying cheaper training costs. As evidence,* - Your citation for Evci et al. (2019) is missing the publication venue/arxiv ID.
assistant
This paper introduces a method to prune networks at initialization in a way that (mostly) preserves the gradient flow through the resulting pruned network. This is a direct improvement over previous methods (e.g. SNIP) which have no guarantees that pruned connections will break the gradient flow and thereby harm learning. I quite like this paper, the motivation and results are convincing and it is well presented. The writing is excellent for most of the paper. From section 5 onwards the writing does need quite a bit of editing, as its quality is significantly reduced from what came before. Some detailed comments: - Figure 1 is very nice and really clarifies the idea! - In paragraph below Equation (8): what does *can be computed by backward twice* mean? - Please specify where the equalities in equation (9) are coming from. - Table 3 & 4: Why are the pruning ratios different for each model? - Table 3: Why are values missing for the baseline for 80% and 90%? - Section 5.2: *We observed that, the main bottleneck or pruned... when deriving the pruning criteria*: it*s not clear where this conclusion is coming from. - Table 5 has no batch size results, even though you*re referencing them in the text. And some minor comments to help with the writing: - Intro: *As shown in Dey et al. (2019) that with pre-specified sparsity, they can achieve* would read better as *As shown by Dey et al. (2019), with pre-specified sparsity one can achieve* - Equation (3): Clarify that this is a function of - Sentence below Equation (6): *of the pruned network, and thus our goal* remove the *and thus* - Table 1: Specify that you*re reporting accuracy. - Section 4.1: *e.g. wide ResNet (Zagaruyko & Komodakis, 2016), and thus we can regard* remove the *and thus* - Sentence below equation (9): *encouraging the eigenspace of Theta align* add a *to* before *align* - Sentence before section 5: *it will encourage the eigenspace of the NTK distributing large eigenvalues in the direction of Y, which will in turn accelerates the decrease of the loss (Arora et al., 2019) and benefits to the optimization in A* would read better as *it will encourage the eigenspace of the NTK to distribute large eigenvalues in the direction of Y, which in turn accelerates the decrease of the loss (Arora et al., 2019) and benefits the optimization in A* - Throughout section 5, write it in present tense rather than past tense. e.g. *In this section, we conduct various experiments* instead of *In this section, we conducted various experiments* - Sentence below table 2: you have *the the* - Second paragraph of section 5.1: *We can observe GraSP outperform random pruning clearly* would read better as *We can observe GraSP clearly outperforms random pruning* - Second paragraph of section 5.1: *In the next, we further compared* remove *In the next* - Second paragraph of section 5.1: *Besides, we further experimented with the late resetting* remove *Besides* - Paragraph above section 5.2: *GraSP surpassing SNIP* use *surpasses* instead - Paragraph above section 5.2: *investigate the reasons behind in Section 5.2 for promoting better understanding* would read better as *investigate the reasons behind this in Section 5.2 for obtaining a better understanding* - Section 5.2: *We observed that, the main bottleneck* -> *We observe that the main bottleneck* - Section 5.2: *Besides, we also plotted the the gradient norm of the pruned*, remove *Besides* and the extra *the* - Section 5.2: *the average of the gradients of the entire dataset* use *over the entire dataset* - Section 5.2: *hopefully more training progress can make as evidenced* would read better as *hopefully more training progress can be made as evidenced* - Section 5.3 title would be better using *Visualizing* instead of *Visualize* - Section 5.3: Join the first two sentences with a comma into a single sentence. - Section 5.3: *In contrast, SNIP are more likely* -> In contrast, SNIP is more likely* - Section 5.4: *for ablation study* would read better as *via ablations* - Section 5.4: *we tested GraSP with three different initialization methods;* use a *:* instead of *;* - Section 6: *Besides, readers may notice that*, remove the *Besides* - Section 6: *traditional pruning algorithms while still enjoy the cheaper training cost. As an evidence,* would read better as *traditional pruning algorithms while still enjoying cheaper training costs. As evidence,* - Your citation for Evci et al. (2019) is missing the publication venue/arxiv ID.