HOW IMPORTANT ARE NETWORK WEIGHTS? TO WHAT EXTENT DO THEY NEED AN UPDATE?
In the context of optimization, a gradient of a neural network indicates the amount a specific weight should change with respect to the loss. Therefore, small gradients indicate a good value of the weight that requires no change and can be kept frozen during training. This paper provides an experimental study on the importance of a neural network weights, and to which extent do they need to be updated. We wish to show that starting from the third epoch, freezing weights which have no informative gradient and are less likely to be changed during training, results in a very slight drop in the overall accuracy (and in sometimes better). We experiment on the MNIST, CIFAR10 and Flickr8k datasets using several architectures (VGG19, ResNet-110 and DenseNet-121). On CIFAR10, we show that freezing 80% of the VGG19 network parameters from the third epoch onwards results in 0.24% drop in accuracy, while freezing 50% of Resnet-110 parameters results in 0.9% drop in accuracy and finally freezing 70% of Densnet-121 parameters results in 0.57% drop in accuracy. Furthermore, to experiemnt with real-life applications, we train an image captioning model with attention mechanism on the Flickr8k dataset using LSTM networks, freezing 60% of the parameters from the third epoch onwards, resulting in a better BLEU-4 score than the fully trained model. Our source code can be found in the appendix.
The paper presents the empirical observation that one can freeze (stop updating) a significant fraction of neural network parameters after only training for a short amount of time, without hurting final performance too much. The technical contribution made by this paper is an algorithm for determining which weights to freeze, called partial backpropagation, and an empirical validation of the algorithm on various models for image recognition. The observation that weights can be frozen is somewhat interesting, although similar findings have been reported before. It*s not clear the proposed algorithm is useful. The authors mention that fully parameterized models are expensive to run, but they don*t demonstrate any speed-ups using their approach. Such speed-up would also not be expected since the forward pass of the algorithm cannot get faster by freezing weights, and the impact on the backward pass is limited. I*d be willing to raise my rating if the authors can convince me of the usefulness of their algorithm.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
HOW IMPORTANT ARE NETWORK WEIGHTS? TO WHAT EXTENT DO THEY NEED AN UPDATE?
In the context of optimization, a gradient of a neural network indicates the amount a specific weight should change with respect to the loss. Therefore, small gradients indicate a good value of the weight that requires no change and can be kept frozen during training. This paper provides an experimental study on the importance of a neural network weights, and to which extent do they need to be updated. We wish to show that starting from the third epoch, freezing weights which have no informative gradient and are less likely to be changed during training, results in a very slight drop in the overall accuracy (and in sometimes better). We experiment on the MNIST, CIFAR10 and Flickr8k datasets using several architectures (VGG19, ResNet-110 and DenseNet-121). On CIFAR10, we show that freezing 80% of the VGG19 network parameters from the third epoch onwards results in 0.24% drop in accuracy, while freezing 50% of Resnet-110 parameters results in 0.9% drop in accuracy and finally freezing 70% of Densnet-121 parameters results in 0.57% drop in accuracy. Furthermore, to experiemnt with real-life applications, we train an image captioning model with attention mechanism on the Flickr8k dataset using LSTM networks, freezing 60% of the parameters from the third epoch onwards, resulting in a better BLEU-4 score than the fully trained model. Our source code can be found in the appendix.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



The paper presents the empirical observation that one can freeze (stop updating) a significant fraction of neural network parameters after only training for a short amount of time, without hurting final performance too much. The technical contribution made by this paper is an algorithm for determining which weights to freeze, called partial backpropagation, and an empirical validation of the algorithm on various models for image recognition. The observation that weights can be frozen is somewhat interesting, although similar findings have been reported before. It*s not clear the proposed algorithm is useful. The authors mention that fully parameterized models are expensive to run, but they don*t demonstrate any speed-ups using their approach. Such speed-up would also not be expected since the forward pass of the algorithm cannot get faster by freezing weights, and the impact on the backward pass is limited. I*d be willing to raise my rating if the authors can convince me of the usefulness of their algorithm.
assistant
The paper presents the empirical observation that one can freeze (stop updating) a significant fraction of neural network parameters after only training for a short amount of time, without hurting final performance too much. The technical contribution made by this paper is an algorithm for determining which weights to freeze, called partial backpropagation, and an empirical validation of the algorithm on various models for image recognition. The observation that weights can be frozen is somewhat interesting, although similar findings have been reported before. It*s not clear the proposed algorithm is useful. The authors mention that fully parameterized models are expensive to run, but they don*t demonstrate any speed-ups using their approach. Such speed-up would also not be expected since the forward pass of the algorithm cannot get faster by freezing weights, and the impact on the backward pass is limited. I*d be willing to raise my rating if the authors can convince me of the usefulness of their algorithm.