Dynamic Sparse Training: Find Efficient Sparse Network From Scratch With Trainable Masked Layers
We present a novel network pruning algorithm called Dynamic Sparse Training that can jointly ?nd the optimal network parameters and sparse network structure in a uni?ed optimization process with trainable pruning thresholds. These thresholds can have ?ne-grained layer-wise adjustments dynamically via backpropagation. We demonstrate that our dynamic sparse training algorithm can easily train very sparse neural network models with little performance loss using the same training epochs as dense models. Dynamic Sparse Training achieves prior art performance compared with other sparse training algorithms on various network architectures. Additionally, we have several surprising observations that provide strong evidence to the effectiveness and ef?ciency of our algorithm. These observations reveal the underlying problems of traditional three-stage pruning algorithms and present the potential guidance provided by our algorithm to the design of more compact network architectures.
## Update after the rebuttal I appreciate the author*s clarification in the rebuttal and the additional result on ImageNet, which addressed some of my concerns. # Summary This paper proposes a trainable mask layer in neural networks for compressing neural networks end-to-end. The main idea is to apply a differentiable mask to individual weights such that the mask itself is also trained through backpropagation. They also propose to add a regularization term that encourages weights are masked out as much as possible. The result on MNIST and CIFAR show that their method can achieve the highest (weight) compression rate and the lowest accuracy reduction compared to baselines. # Originality - The idea of applying trainable mask to weights and regularizing toward masking out is quite interesting and new to my knowledge. # Quality - The performance seems to be good, though it would be more convincing if the paper showed results on larger datasets like ImageNet. - The analysis is interesting, but I am not fully convinced by the *strong evidence to the efficiency and effectiveness of our algorithm*. For example, the final layer*s remaining ratio is constantly 1 in Figure 3, while it starts from nearly 0 in Figure 4. The paper also argues that the final layer was not that important in Figure 4 because the lower layers have not learned useful features. This seems not only contradictory to the result of Figure 3 but also inconsistent of the accuracy being quickly increasing up to near 90% while the remaining ratio is nearly 0 in Figure 4. - If the motivation of the sparse training is to reduce memory consumption AND computation, showing some results on the reduction of the computation cost after sparse training would important to complete the story. # Clarity - The description of the main idea is not clear. - What are *structure gradient* and *performance gradient*? They are not mathematically defined in the paper. - I do not understand how the proposed method can *recover* from pruned connection, although it seems to be indeed happening in the experiment. The paper claims that the use of long-tailed higher-order estimator H(x) makes it possible to recover. However, H(x) still seems to have flat lines where the derivative is 0. Is H(x) in Equation 3 and Figure 2d are showing *derivative* or step function itself? In any cases, I do not see how the gradient flows once a weight is masked out. # Significance - This paper proposes an interesting idea (trainable mask), though I did not fully get how the mask is defined/trained and has a potential to recover after pruning. The analysis of the compression rate throughout training is interesting but does not seem to be fully convincing. It would be stronger if the paper 1) included more results on bigger datasets like ImageNet, 2) described the main idea more clearly, and 3) provided more convincing evidence why the proposed method is effective.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Dynamic Sparse Training: Find Efficient Sparse Network From Scratch With Trainable Masked Layers
We present a novel network pruning algorithm called Dynamic Sparse Training that can jointly ?nd the optimal network parameters and sparse network structure in a uni?ed optimization process with trainable pruning thresholds. These thresholds can have ?ne-grained layer-wise adjustments dynamically via backpropagation. We demonstrate that our dynamic sparse training algorithm can easily train very sparse neural network models with little performance loss using the same training epochs as dense models. Dynamic Sparse Training achieves prior art performance compared with other sparse training algorithms on various network architectures. Additionally, we have several surprising observations that provide strong evidence to the effectiveness and ef?ciency of our algorithm. These observations reveal the underlying problems of traditional three-stage pruning algorithms and present the potential guidance provided by our algorithm to the design of more compact network architectures.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



## Update after the rebuttal I appreciate the author*s clarification in the rebuttal and the additional result on ImageNet, which addressed some of my concerns. # Summary This paper proposes a trainable mask layer in neural networks for compressing neural networks end-to-end. The main idea is to apply a differentiable mask to individual weights such that the mask itself is also trained through backpropagation. They also propose to add a regularization term that encourages weights are masked out as much as possible. The result on MNIST and CIFAR show that their method can achieve the highest (weight) compression rate and the lowest accuracy reduction compared to baselines. # Originality - The idea of applying trainable mask to weights and regularizing toward masking out is quite interesting and new to my knowledge. # Quality - The performance seems to be good, though it would be more convincing if the paper showed results on larger datasets like ImageNet. - The analysis is interesting, but I am not fully convinced by the *strong evidence to the efficiency and effectiveness of our algorithm*. For example, the final layer*s remaining ratio is constantly 1 in Figure 3, while it starts from nearly 0 in Figure 4. The paper also argues that the final layer was not that important in Figure 4 because the lower layers have not learned useful features. This seems not only contradictory to the result of Figure 3 but also inconsistent of the accuracy being quickly increasing up to near 90% while the remaining ratio is nearly 0 in Figure 4. - If the motivation of the sparse training is to reduce memory consumption AND computation, showing some results on the reduction of the computation cost after sparse training would important to complete the story. # Clarity - The description of the main idea is not clear. - What are *structure gradient* and *performance gradient*? They are not mathematically defined in the paper. - I do not understand how the proposed method can *recover* from pruned connection, although it seems to be indeed happening in the experiment. The paper claims that the use of long-tailed higher-order estimator H(x) makes it possible to recover. However, H(x) still seems to have flat lines where the derivative is 0. Is H(x) in Equation 3 and Figure 2d are showing *derivative* or step function itself? In any cases, I do not see how the gradient flows once a weight is masked out. # Significance - This paper proposes an interesting idea (trainable mask), though I did not fully get how the mask is defined/trained and has a potential to recover after pruning. The analysis of the compression rate throughout training is interesting but does not seem to be fully convincing. It would be stronger if the paper 1) included more results on bigger datasets like ImageNet, 2) described the main idea more clearly, and 3) provided more convincing evidence why the proposed method is effective.
assistant
## Update after the rebuttal I appreciate the author*s clarification in the rebuttal and the additional result on ImageNet, which addressed some of my concerns. # Summary This paper proposes a trainable mask layer in neural networks for compressing neural networks end-to-end. The main idea is to apply a differentiable mask to individual weights such that the mask itself is also trained through backpropagation. They also propose to add a regularization term that encourages weights are masked out as much as possible. The result on MNIST and CIFAR show that their method can achieve the highest (weight) compression rate and the lowest accuracy reduction compared to baselines. # Originality - The idea of applying trainable mask to weights and regularizing toward masking out is quite interesting and new to my knowledge. # Quality - The performance seems to be good, though it would be more convincing if the paper showed results on larger datasets like ImageNet. - The analysis is interesting, but I am not fully convinced by the *strong evidence to the efficiency and effectiveness of our algorithm*. For example, the final layer*s remaining ratio is constantly 1 in Figure 3, while it starts from nearly 0 in Figure 4. The paper also argues that the final layer was not that important in Figure 4 because the lower layers have not learned useful features. This seems not only contradictory to the result of Figure 3 but also inconsistent of the accuracy being quickly increasing up to near 90% while the remaining ratio is nearly 0 in Figure 4. - If the motivation of the sparse training is to reduce memory consumption AND computation, showing some results on the reduction of the computation cost after sparse training would important to complete the story. # Clarity - The description of the main idea is not clear. - What are *structure gradient* and *performance gradient*? They are not mathematically defined in the paper. - I do not understand how the proposed method can *recover* from pruned connection, although it seems to be indeed happening in the experiment. The paper claims that the use of long-tailed higher-order estimator H(x) makes it possible to recover. However, H(x) still seems to have flat lines where the derivative is 0. Is H(x) in Equation 3 and Figure 2d are showing *derivative* or step function itself? In any cases, I do not see how the gradient flows once a weight is masked out. # Significance - This paper proposes an interesting idea (trainable mask), though I did not fully get how the mask is defined/trained and has a potential to recover after pruning. The analysis of the compression rate throughout training is interesting but does not seem to be fully convincing. It would be stronger if the paper 1) included more results on bigger datasets like ImageNet, 2) described the main idea more clearly, and 3) provided more convincing evidence why the proposed method is effective.