A Simple yet Effective Method to Prune Dense Layers of Neural Networks
Neural networks are usually over-parameterized with significant redundancy in the number of required neurons which results in unnecessary computation and memory usage at inference time. One common approach to address this issue is to prune these big networks by removing extra neurons and parameters while maintaining the accuracy. In this paper, we propose NoiseOut, a fully automated pruning algorithm based on the correlation between activations of neurons in the hidden layers. We prove that adding additional output neurons with entirely random targets results into a higher correlation between neurons which makes pruning by NoiseOut even more efficient. Finally, we test our method on various networks and datasets. These experiments exhibit high pruning rates while maintaining the accuracy of the original network.
This paper proposes and tests two ideas. (1) a method of pruning networks by identifying highly correlated neuron pairs, pruning one of the pair, and then modifying downstream weights to compensate for the removal (which works well if the removed neurons were highly correlated). (2) a method, dubbed NoiseOut, for increasing neuron correlation by adding auxiliary noise target outputs to the network during training. The first idea (1) is fairly straightforward, and it is not clear if it has been tried before. It does seem to work. The second idea (2) is of unclear value and seems to this reviewer that it may merely add a regularizing effect. Comments in this direction: - In Fig 4 (right), the constant and Gaussian treatments seem to produce the same effect in both networks, right? And the Binomial effect seems the same as No_Noise. If this is true, can we conclude that the NoiseOut targets are simply serving to regularize the network, that is, to reduce its capacity slightly? - To show whether this effect is true, one would need to compare to other methods of reducing the network capacity, for example: by reducing the number of neurons, by applying L2 regularization of various values, or by applying Dropout of various strengths. Fig 7 makes an attempt at this direction, but critically misses several comparison treatments: “Pruned without any regularization”, “Pruned with only L2”, and “Pruned with only DropOut”. Have these experiments been run? Can their results be included and used to produce plots like Fig 5 and Fig 7? Without these comparisons, it seems impossible to conclude that NoiseOut does anything but provide similar regularization to DropOut or L2. The combined ideas (1) + (2) DO produce a considerable reduction in parameters, but sadly the experiments and exposition are somewhat too lacking to really understand what is going on. With a little more work the paper could be quite interesting, but as is it should probably not be accepted. Additional comments: - Section 4 states: “In all of these experiments, the only stop criteria is the accuracy decay of the model. We set the threshold for this criteria to match the original accuracy; therefore all the compressed network have the same accuracy as the original network.” Is this accuracy the train accuracy or test accuracy? If train, then test accuracy needs to be shown (how much test performance is lost when pruning?). If test, then this would typically be referred to as “cheating” and so the choice needs to be very clearly stated and then defended. - Lowercase rho is used to indicate correlation but this is never actually specified, which is confusing for. Just state once that it indicates correlation. - How do these results compare to other pruning methods? No numerical comparison is attempted.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
A Simple yet Effective Method to Prune Dense Layers of Neural Networks
Neural networks are usually over-parameterized with significant redundancy in the number of required neurons which results in unnecessary computation and memory usage at inference time. One common approach to address this issue is to prune these big networks by removing extra neurons and parameters while maintaining the accuracy. In this paper, we propose NoiseOut, a fully automated pruning algorithm based on the correlation between activations of neurons in the hidden layers. We prove that adding additional output neurons with entirely random targets results into a higher correlation between neurons which makes pruning by NoiseOut even more efficient. Finally, we test our method on various networks and datasets. These experiments exhibit high pruning rates while maintaining the accuracy of the original network.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This paper proposes and tests two ideas. (1) a method of pruning networks by identifying highly correlated neuron pairs, pruning one of the pair, and then modifying downstream weights to compensate for the removal (which works well if the removed neurons were highly correlated). (2) a method, dubbed NoiseOut, for increasing neuron correlation by adding auxiliary noise target outputs to the network during training. The first idea (1) is fairly straightforward, and it is not clear if it has been tried before. It does seem to work. The second idea (2) is of unclear value and seems to this reviewer that it may merely add a regularizing effect. Comments in this direction: - In Fig 4 (right), the constant and Gaussian treatments seem to produce the same effect in both networks, right? And the Binomial effect seems the same as No_Noise. If this is true, can we conclude that the NoiseOut targets are simply serving to regularize the network, that is, to reduce its capacity slightly? - To show whether this effect is true, one would need to compare to other methods of reducing the network capacity, for example: by reducing the number of neurons, by applying L2 regularization of various values, or by applying Dropout of various strengths. Fig 7 makes an attempt at this direction, but critically misses several comparison treatments: “Pruned without any regularization”, “Pruned with only L2”, and “Pruned with only DropOut”. Have these experiments been run? Can their results be included and used to produce plots like Fig 5 and Fig 7? Without these comparisons, it seems impossible to conclude that NoiseOut does anything but provide similar regularization to DropOut or L2. The combined ideas (1) + (2) DO produce a considerable reduction in parameters, but sadly the experiments and exposition are somewhat too lacking to really understand what is going on. With a little more work the paper could be quite interesting, but as is it should probably not be accepted. Additional comments: - Section 4 states: “In all of these experiments, the only stop criteria is the accuracy decay of the model. We set the threshold for this criteria to match the original accuracy; therefore all the compressed network have the same accuracy as the original network.” Is this accuracy the train accuracy or test accuracy? If train, then test accuracy needs to be shown (how much test performance is lost when pruning?). If test, then this would typically be referred to as “cheating” and so the choice needs to be very clearly stated and then defended. - Lowercase rho is used to indicate correlation but this is never actually specified, which is confusing for. Just state once that it indicates correlation. - How do these results compare to other pruning methods? No numerical comparison is attempted.
assistant
This paper proposes and tests two ideas. (1) a method of pruning networks by identifying highly correlated neuron pairs, pruning one of the pair, and then modifying downstream weights to compensate for the removal (which works well if the removed neurons were highly correlated). (2) a method, dubbed NoiseOut, for increasing neuron correlation by adding auxiliary noise target outputs to the network during training. The first idea (1) is fairly straightforward, and it is not clear if it has been tried before. It does seem to work. The second idea (2) is of unclear value and seems to this reviewer that it may merely add a regularizing effect. Comments in this direction: - In Fig 4 (right), the constant and Gaussian treatments seem to produce the same effect in both networks, right? And the Binomial effect seems the same as No_Noise. If this is true, can we conclude that the NoiseOut targets are simply serving to regularize the network, that is, to reduce its capacity slightly? - To show whether this effect is true, one would need to compare to other methods of reducing the network capacity, for example: by reducing the number of neurons, by applying L2 regularization of various values, or by applying Dropout of various strengths. Fig 7 makes an attempt at this direction, but critically misses several comparison treatments: “Pruned without any regularization”, “Pruned with only L2”, and “Pruned with only DropOut”. Have these experiments been run? Can their results be included and used to produce plots like Fig 5 and Fig 7? Without these comparisons, it seems impossible to conclude that NoiseOut does anything but provide similar regularization to DropOut or L2. The combined ideas (1) + (2) DO produce a considerable reduction in parameters, but sadly the experiments and exposition are somewhat too lacking to really understand what is going on. With a little more work the paper could be quite interesting, but as is it should probably not be accepted. Additional comments: - Section 4 states: “In all of these experiments, the only stop criteria is the accuracy decay of the model. We set the threshold for this criteria to match the original accuracy; therefore all the compressed network have the same accuracy as the original network.” Is this accuracy the train accuracy or test accuracy? If train, then test accuracy needs to be shown (how much test performance is lost when pruning?). If test, then this would typically be referred to as “cheating” and so the choice needs to be very clearly stated and then defended. - Lowercase rho is used to indicate correlation but this is never actually specified, which is confusing for. Just state once that it indicates correlation. - How do these results compare to other pruning methods? No numerical comparison is attempted.